SAT: unit tests `test_state_with_abnormally_large_values` + add missing `future_state_path` #17791

alafanechere · 2022-10-10T14:54:02Z

What

Closes #17338
Just a sanity check over our "source should always emit state" testing.

How

Unit test the test_state_with_abnormally_large_values to make sure it works as expected
Add missing future_state_path to acceptance-test-config.yml of the following connectors:
- source-surveymonkey [GA]
- source-bigcommerce [alpha]
- source-talkdesk-explore [alpha]
- source-paystack [alpha]

The other connector missing future_state_path testing are the following:

source-amplitude [GA]
source-snapchat-marketing [GA]
source-zendesk-chat [GA]
source-aws-cloudtrail [alpha]
source-strava [alpha]

It's not possible to check these connector always emit a state when no record is available because their API does not allow querying future data, which is what test_state_with_abnormally_large_values is testing.

github-actions · 2022-10-10T14:54:55Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-chargebee
source-asana
source-sentry
source-gitlab
source-braintree
source-pinterest
source-okta
source-lemlist
source-google-ads
source-cart
source-zenloop
source-openweather
source-notion
source-freshcaller
source-delighted
source-freshsales
source-azure-table
source-strava
source-harvest
source-lever-hiring
source-klaviyo
source-pipedrive
source-retently
source-commercetools
source-instagram
source-youtube-analytics
source-salesforce
source-onesignal
source-plaid
source-facebook-marketing
source-appsflyer
source-google-search-console
source-amplitude
source-amazon-ads
source-facebook-pages
source-outreach
source-posthog
source-recharge
source-prestashop
source-github
source-drift
source-surveymonkey
source-greenhouse
source-linnworks
source-confluence
source-airtable
source-sendgrid
source-zendesk-talk
source-amazon-sqs
source-zendesk-sunshine
source-iterable
source-twilio
source-pardot
source-mailgun
source-paystack
source-amazon-seller-partner
source-salesloft
source-mailchimp
source-monday
source-freshservice
source-tplcentral
source-quickbooks-singer

alafanechere · 2022-10-10T15:04:29Z

/test connector=connectors/source-surveymonkey

🕑 connectors/source-surveymonkey https://github.com/airbytehq/airbyte/actions/runs/3220346873
✅ connectors/source-surveymonkey https://github.com/airbytehq/airbyte/actions/runs/3220346873
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_surveymonkey/__init__.py       2      0   100%
source_surveymonkey/streams.py      131      7    95%
source_surveymonkey/source.py        49     13    73%
-----------------------------------------------------
TOTAL                               182     20    89%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     20    87%   21-23, 29-31, 36-43, 48-61, 239
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    453    67%

Build Passed

Test summary info:

All Passed

alafanechere · 2022-10-10T15:04:37Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/3220347370
❌ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/3220347370
🐛 https://gradle.com/s/nmt3wsu3c7za6

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
======================== 1 failed, 28 passed in 56.66s =========================

alafanechere · 2022-10-10T15:04:52Z

/test connector=connectors/source-talkdesk-explore

🕑 connectors/source-talkdesk-explore https://github.com/airbytehq/airbyte/actions/runs/3220350062
❌ connectors/source-talkdesk-explore https://github.com/airbytehq/airbyte/actions/runs/3220350062
🐛 https://gradle.com/s/ygd3rxbsk27oo

Build Failed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
============================== 1 skipped in 1.32s ==============================

alafanechere · 2022-10-10T15:05:02Z

/test connector=connectors/source-paystack

🕑 connectors/source-paystack https://github.com/airbytehq/airbyte/actions/runs/3220350074
❌ connectors/source-paystack https://github.com/airbytehq/airbyte/actions/runs/3220350074
🐛 https://gradle.com/s/k5gecf34dn3o2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_additional_properties_is_true[inputs0] - ...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
=================== 2 failed, 27 passed in 81.46s (0:01:21) ====================

github-actions · 2022-10-10T15:05:02Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-gitlab
source-linnworks
source-youtube-analytics
source-zendesk-sunshine
source-freshcaller
source-cart
source-chargebee
source-delighted
source-pipedrive
source-sentry
source-salesforce
source-freshsales
source-appsflyer
source-freshservice
source-recharge
source-amplitude
source-sendgrid
source-drift
source-google-search-console
source-lever-hiring
source-greenhouse
source-salesloft
source-strava
source-google-ads
source-instagram
source-facebook-pages
source-github
source-facebook-marketing
source-lemlist
source-amazon-sqs
source-confluence
source-retently
source-amazon-ads
source-zenloop
source-commercetools
source-mailgun
source-okta
source-monday
source-paystack
source-onesignal
source-zendesk-talk
source-tplcentral
source-braintree
source-posthog
source-amazon-seller-partner
source-harvest
source-azure-table
source-quickbooks-singer
source-asana
source-prestashop
source-pinterest
source-iterable
source-notion
source-twilio
source-surveymonkey
source-mailchimp
source-airtable
source-klaviyo
source-pardot
source-openweather
source-outreach
source-plaid

alafanechere · 2022-10-10T15:07:40Z

I don't expect SAT to pass on alpha connectors are they are also failing on master...

evantahler

👍

A question about "The other connector missing future_state_path testing are the following"

did you audit all the connectors? How did you come up with this list?

A question about: "I don't expect SAT to pass on alpha connectors are they are also failing on master..."

How did you learn this? I wonder if we should keep a spreadsheet or metabase report about this to help connector developers know what to do if they need to re-publish a connector

evantahler · 2022-10-10T16:41:53Z

airbyte-integrations/connectors/source-bigcommerce/acceptance-test-config.yml

@@ -18,7 +18,7 @@ tests:
  incremental:
    - config_path: "secrets/config.json"
      configured_catalog_path: "integration_tests/configured_catalog.json"
-      # future_state_path: "integration_tests/abnormal_state.json"
+      future_state_path: "integration_tests/abnormal_state.json"


I love to see more tests being enabled!

alafanechere · 2022-10-10T16:54:18Z

did you audit all the connectors? How did you come up with this list?

Quick script parsing the acceptance-test-config.yml files of each connector declaring incremental test. Declared missing if future_state_path is not set.
I'd like to suggest a permanent approach at this coverage evaluation in the tech spec of #17531

A question about: "I don't expect SAT to pass on alpha connectors are they are also failing on master..."
How did you learn this? I wonder if we should keep a spreadsheet or metabase report about this to help connector developers know what to do if they need to re-publish a connector

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/builds.md

alafanechere · 2022-10-11T10:18:23Z

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3226018106

Connector	Did it publish?	Were definitions generated?
bases/source-acceptance-test	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2022-10-11T17:35:34Z

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3228970392

Connector	Did it publish?	Were definitions generated?
bases/source-acceptance-test	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-11T17:36:16Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-paystack
source-github
source-confluence
source-notion
source-harvest
source-mailgun
source-gitlab
source-azure-table
source-pardot
source-zendesk-sunshine
source-asana
source-klaviyo
source-braintree
source-tplcentral
source-sentry
source-amazon-ads
source-instagram
source-airtable
source-zendesk-talk
source-okta
source-iterable
source-recharge
source-zenloop
source-prestashop
source-pinterest
source-youtube-analytics
source-drift
source-delighted
source-pipedrive
source-onesignal
source-lever-hiring
source-freshcaller
source-openweather
source-retently
source-posthog
source-salesforce
source-twilio
source-amazon-seller-partner
source-commercetools
source-plaid
source-freshsales
source-cart
source-salesloft
source-amplitude
source-facebook-marketing
source-lemlist
source-facebook-pages
source-appsflyer
source-freshservice
source-mailchimp
source-chargebee
source-google-search-console
source-outreach
source-quickbooks-singer
source-surveymonkey
source-linnworks
source-amazon-sqs
source-monday
source-strava
source-google-ads
source-sendgrid
source-greenhouse

alafanechere · 2022-10-12T10:19:00Z

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3233879029

Connector	Did it publish?	Were definitions generated?
bases/source-acceptance-test	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-12T10:19:32Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-salesforce
source-zendesk-talk
source-braintree
source-posthog
source-facebook-marketing
source-asana
source-linnworks
source-pardot
source-amazon-sqs
source-confluence
source-drift
source-amazon-seller-partner
source-notion
source-sentry
source-mailgun
source-twilio
source-tplcentral
source-amplitude
source-quickbooks-singer
source-google-ads
source-youtube-analytics
source-instagram
source-recharge
source-klaviyo
source-plaid
source-outreach
source-freshsales
source-retently
source-github
source-freshservice
source-okta
source-amazon-ads
source-zenloop
source-chargebee
source-commercetools
source-pinterest
source-google-search-console
source-paystack
source-facebook-pages
source-appsflyer
source-mailchimp
source-cart
source-harvest
source-salesloft
source-sendgrid
source-zendesk-sunshine
source-lemlist
source-prestashop
source-greenhouse
source-delighted
source-pipedrive
source-airtable
source-monday
source-gitlab
source-strava
source-azure-table
source-lever-hiring
source-surveymonkey
source-openweather
source-freshcaller
source-iterable
source-onesignal

alafanechere · 2022-10-12T15:39:15Z

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3236002361

Connector	Did it publish?	Were definitions generated?
bases/source-acceptance-test	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-12T15:40:06Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-confluence
source-klaviyo
source-salesforce
source-iterable
source-pardot
source-freshsales
source-airtable
source-pinterest
source-appsflyer
source-zenloop
source-salesloft
source-sendgrid
source-surveymonkey
source-amazon-seller-partner
source-greenhouse
source-google-search-console
source-paystack
source-asana
source-delighted
source-lemlist
source-linnworks
source-facebook-pages
source-sentry
source-notion
source-posthog
source-lever-hiring
source-cart
source-strava
source-youtube-analytics
source-facebook-marketing
source-mailgun
source-chargebee
source-zendesk-sunshine
source-prestashop
source-mailchimp
source-outreach
source-commercetools
source-monday
source-drift
source-freshcaller
source-twilio
source-amazon-ads
source-braintree
source-gitlab
source-google-ads
source-onesignal
source-tplcentral
source-plaid
source-retently
source-quickbooks-singer
source-azure-table
source-harvest
source-recharge
source-freshservice
source-amazon-sqs
source-zendesk-talk
source-okta
source-pipedrive
source-amplitude
source-github
source-openweather
source-instagram

…ng `future_state_path` (airbytehq#17791)

alafanechere added 3 commits October 10, 2022 16:02

test state emit when no data

fb250c3

bump version

918352e

add abnormal state test to when missing

4c4d9fb

github-actions bot added the area/connectors Connector related issues label Oct 10, 2022

octavia-squidington-iv added connectors/source/surveymonkey connectors/source/bigcommerce connectors/source/paystack connectors/source/talkdesk-explore labels Oct 10, 2022

Merge branch 'master' into augustin/sat/allways-emit-state

0ce9b2f

alafanechere marked this pull request as ready for review October 10, 2022 15:04

alafanechere requested a review from evantahler October 10, 2022 15:05

evantahler approved these changes Oct 10, 2022

View reviewed changes

Merge branch 'master' into augustin/sat/allways-emit-state

da3902f

Merge branch 'master' into augustin/sat/allways-emit-state

701c9c0

Merge branch 'master' into augustin/sat/allways-emit-state

7ce455b

alafanechere merged commit 0cbd4de into master Oct 12, 2022

alafanechere deleted the augustin/sat/allways-emit-state branch October 12, 2022 17:32

This was referenced Oct 13, 2022

Bump Airbyte version from 0.40.14 to 0.40.15 #17961

Closed

Bump Airbyte version from 0.40.14 to 0.40.15 #17970

Merged

jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022

SAT: unit tests test_state_with_abnormally_large_values + add missi…

2a1b3bd

…ng `future_state_path` (airbytehq#17791)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SAT: unit tests `test_state_with_abnormally_large_values` + add missing `future_state_path` #17791

SAT: unit tests `test_state_with_abnormally_large_values` + add missing `future_state_path` #17791

alafanechere commented Oct 10, 2022 •

edited

Loading

github-actions bot commented Oct 10, 2022

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 10, 2022

alafanechere commented Oct 10, 2022

evantahler left a comment •

edited

Loading

evantahler Oct 10, 2022

alafanechere commented Oct 10, 2022

alafanechere commented Oct 11, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 11, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 11, 2022

alafanechere commented Oct 12, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 12, 2022

alafanechere commented Oct 12, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 12, 2022

SAT: unit tests test_state_with_abnormally_large_values + add missing future_state_path #17791

SAT: unit tests test_state_with_abnormally_large_values + add missing future_state_path #17791

Conversation

alafanechere commented Oct 10, 2022 • edited Loading

What

How

github-actions bot commented Oct 10, 2022

alafanechere commented Oct 10, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere commented Oct 10, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Oct 10, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Oct 10, 2022 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Oct 10, 2022

alafanechere commented Oct 10, 2022

evantahler left a comment • edited Loading

Choose a reason for hiding this comment

evantahler Oct 10, 2022

Choose a reason for hiding this comment

alafanechere commented Oct 10, 2022

alafanechere commented Oct 11, 2022 • edited by github-actions bot Loading

alafanechere commented Oct 11, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 11, 2022

alafanechere commented Oct 12, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 12, 2022

alafanechere commented Oct 12, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 12, 2022

SAT: unit tests `test_state_with_abnormally_large_values` + add missing `future_state_path` #17791

SAT: unit tests `test_state_with_abnormally_large_values` + add missing `future_state_path` #17791

alafanechere commented Oct 10, 2022 •

edited

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 10, 2022 •

edited by github-actions bot

Loading

evantahler left a comment •

edited

Loading

alafanechere commented Oct 11, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 11, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 12, 2022 •

edited by github-actions bot

Loading

alafanechere commented Oct 12, 2022 •

edited by github-actions bot

Loading