Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛Source Amazon Seller Partner: Implement slicing for report streams #16430

Merged

Conversation

davydov-d
Copy link
Collaborator

@davydov-d davydov-d commented Sep 8, 2022

What

https://github.com/airbytehq/alpha-beta-issues/issues/257

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 8, 2022
@davydov-d
Copy link
Collaborator Author

davydov-d commented Sep 8, 2022

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3013690534
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3013690534
🐛 https://gradle.com/s/m6fxtyfru6fmu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 2 failed, 20 passed, 3 skipped in 18.85s ===================

@davydov-d
Copy link
Collaborator Author

davydov-d commented Sep 8, 2022

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3014141430
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3014141430
🐛 https://gradle.com/s/acdgstzsiw56k

Build Failed

Test summary info:

Could not find result summary

@davydov-d
Copy link
Collaborator Author

davydov-d commented Sep 8, 2022

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3014422589
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3014422589
🐛 https://gradle.com/s/3dsmzklyg2hwk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >300.0s
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs2] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs3] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs5] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs6] - AssertionError: At l...
FAILED test_core.py::TestBasicRead::test_read[inputs7] - Failed: Timeout >300.0s
FAILED test_core.py::TestBasicRead::test_read[inputs8] - AssertionError: At l...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
ERROR test_core.py::TestBasicRead::test_read[inputs4] - FileNotFoundError: [E...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:57: Backward compatibility tests are disabled for version 0.2.25.
======== 12 failed, 31 passed, 1 skipped, 1 error in 3970.56s (1:06:10) ========

@davydov-d davydov-d marked this pull request as draft October 19, 2022 07:15
@davydov-d davydov-d linked an issue Oct 19, 2022 that may be closed by this pull request
@animer3009
Copy link
Contributor

animer3009 commented Nov 4, 2022

@davydov-d
@bazarnov

Are you sure that it can't be more than 30 days?

Some reports can requested with 60 day period.

For example you can find You can request up to 60 days of data in a single report in descriptions here:

https://developer-docs.amazon.com/sp-api/docs/report-type-values

So please do not restrict period_in_days.

@davydov-d
Copy link
Collaborator Author

@davydov-d @bazarnov

Are you sure that it can't be more than 30 days?

Some reports can requested with 60 day period.

For example you can find You can request up to 60 days of data in a single report in descriptions here:

https://developer-docs.amazon.com/sp-api/docs/report-type-values

So please do not restrict period_in_days.

@animer3009 well I can't be sure of anything since this is Amazon API 😅
thanks for your feedback, I guess we'll need to do a more intense research

@lazebnyi lazebnyi changed the title Source amazon seller partner: implement slicing for report streams 🐛Source Amazon Seller Partner: Implement slicing for report streams Dec 7, 2022
@lazebnyi
Copy link
Collaborator

lazebnyi commented Dec 16, 2022

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3716216122
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3716216122
🐛 https://gradle.com/s/bvc46cnqlimqg

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >300.0s
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs2] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs3] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs5] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs6] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs7] - Failed: Timeout >300.0s
FAILED test_core.py::TestBasicRead::test_read[inputs8] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
ERROR test_core.py::TestBasicRead::test_read[inputs4] - FileNotFoundError: [E...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
= 14 failed, 31 passed, 1 skipped, 31 warnings, 1 error in 3061.55s (0:51:01) ==

> Task :airbyte-integrations:connectors:source-amazon-seller-partner:sourceAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
48 actionable tasks: 23 executed, 25 up-to-date

Publishing build scan...
https://gradle.com/s/bvc46cnqlimqg

The remote build cache was disabled during the build due to errors.
S3 cache writes: 1, elapsed: 627ms, sent to cache: 446 B

@lazebnyi lazebnyi self-assigned this Dec 16, 2022
@animer3009
Copy link
Contributor

@lazebnyi @bazarnov
Guys are you anyway trying to push changes to restrict period_in_days with 30?
Have you seen my comment?
#16430 (comment)

@lazebnyi
Copy link
Collaborator

@lazebnyi @bazarnov

Guys are you anyway trying to push changes to restrict period_in_days with 30?

Have you seen my comment?

#16430 (comment)

Yes, we saw your comments. Thanks for that🙏 We will try implement different rages for different streams. That's not final version of PR. Just draft 😊

@lazebnyi lazebnyi marked this pull request as ready for review January 11, 2023 18:46
@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 11, 2023

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3895749684
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3895749684
Python tests coverage:

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/spec.py           23      0   100%
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         51     12    76%
source_amazon_seller_partner/streams.py       586    255    56%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         760    304    60%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   56-63, 68-81, 244
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.30.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 0.2.30.
================== 29 passed, 2 skipped in 1184.18s (0:19:44) ==================

@lazebnyi lazebnyi requested review from alafanechere and removed request for grubberr January 11, 2023 19:49
@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 18, 2023

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3953427100
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3953427100
🐛

@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 19, 2023

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3958419728
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3958419728
Python tests coverage:

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/spec.py           23      0   100%
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         51     12    76%
source_amazon_seller_partner/streams.py       586    255    56%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         760    304    60%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.30.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 0.2.30.
================== 29 passed, 2 skipped in 1216.96s (0:20:16) ==================

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The acceptance-test-config.ymlfile looks good to me now that the test_strictness_level is not set to high.
You are introducing a small backward incompatible change:
Users with configuration with the period_in_days to a value > 30 will likely get a check command failure.
Please sync up with @airbytehq/cloud-support before merging to make sure outreach is sent to the customer using this connector.

@alafanechere alafanechere requested a review from a team January 23, 2023 17:53
@tybernstein
Copy link
Contributor

@alafanechere I just checked and there is only one workspace on Cloud that has this connector setup and it is internal (@amaliaroye) so there is no outreach necessary from TCS.

Copy link
Contributor

@tybernstein tybernstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No outreach necessary as there are no Cloud users impacted by this update.

@animer3009
Copy link
Contributor

animer3009 commented Jan 23, 2023

Guys,
You are still going to merge changes which restrict period_in_days with 30 day.
I will repeat the same:

  1. Some reports allow 30 day +
  2. We are using Airbyte SP for the production. As soon as you will merge this, we will face issue! As sometimes we are requesting 31 and 30 day + reports.

So the main question is:
Why you are not keeping possibility to request 30 day + reports. Just add a notice that recommended use 30! Or restrict it just for cloud...

@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 26, 2023

@animer3009 Thanks! We decide increase max value for period_in_days to 90 days. But for GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL and GET_FLAT_FILE_RETURNS_DATA_BY_RETURN_DATE streams maximum value for period_in_days 30 days and 60 days.
So, for any value that exceeds the limit, the period_in_days will be automatically reduced to the limit for the stream.

@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 26, 2023

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4016842466
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4016842466
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.30.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 0.2.30.
================== 29 passed, 2 skipped in 2231.43s (0:37:11) ==================

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4016842466
❌ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4016842466
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.30.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 0.2.30.
================== 29 passed, 2 skipped in 1449.76s (0:24:09) ==================

@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 26, 2023

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4017585312
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/4017585312
Python tests coverage:

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/spec.py           23      0   100%
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         51     12    76%
source_amazon_seller_partner/streams.py       587    255    57%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         761    304    60%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.30.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 0.2.30.
================== 29 passed, 2 skipped in 1179.93s (0:19:39) ==================

@lazebnyi
Copy link
Collaborator

lazebnyi commented Jan 26, 2023

/publish connector=connectors/source-amazon-seller-partner

🕑 Publishing the following connectors:
connectors/source-amazon-seller-partner
https://github.com/airbytehq/airbyte/actions/runs/4017831069


Connector Did it publish? Were definitions generated?
connectors/source-amazon-seller-partner

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@octavia-squidington-iv octavia-squidington-iv removed the area/frontend Related to the Airbyte webapp label Jan 26, 2023
@lazebnyi lazebnyi temporarily deployed to more-secrets January 26, 2023 18:56 — with GitHub Actions Inactive
@lazebnyi lazebnyi temporarily deployed to more-secrets January 26, 2023 18:56 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage 23.99%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/amazon-seller-partner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Amazon Seller Partner (Cloud Only) Connection Error