Source twilio: fix generating date ranges #19479

davydov-d · 2022-11-16T14:19:12Z

What

https://github.com/airbytehq/oncall/issues/1046

How

When generating datetime ranges we should take into account the cursor field type - in case it is of date type, the stream used to yield extra slice where start date was greater than end date. Now this is fixed

davydov-d · 2022-11-16T14:20:37Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3480174976
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3480174976
🐛 https://gradle.com/s/2u66lzuqqnwga

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 ERROR unit_tests/test_streams.py
	 !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
	 �[31m=============================== �[31m�[1m1 error�[0m�[31m in 1.56s�[0m�[31m ===============================�[0m

davydov-d · 2022-11-16T18:41:53Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3482165597
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3482165597
🐛 https://gradle.com/s/5ygly7qgnrwxc

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs1]
================== 32 passed, 2 errors in 1373.59s (0:22:53) ===================

davydov-d · 2022-11-21T09:56:51Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3513325524
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3513325524
🐛 https://gradle.com/s/tmzjqylid2ewq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream avail...
================== 1 failed, 33 passed in 1097.89s (0:18:17) ===================

davydov-d · 2022-11-21T10:41:30Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3513657348
✅ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3513657348
Python tests coverage:

Name                        Stmts   Miss  Cover
-----------------------------------------------
source_twilio/auth.py           8      0   100%
source_twilio/__init__.py       2      0   100%
source_twilio/source.py        28      2    93%
source_twilio/streams.py      279     21    92%
-----------------------------------------------
TOTAL                         317     23    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

All Passed

davydov-d · 2022-11-23T08:48:19Z

/test connector=connectors/source-twilio

davydov-d · 2022-11-23T10:14:41Z

/test connector=connectors/source-twilio

davydov-d · 2022-11-23T10:42:59Z

/test connector=connectors/source-twilio

davydov-d · 2022-11-28T08:58:42Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3563209655
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3563209655
🐛 https://gradle.com/s/mdjn3q54xfarm

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream avail...
================== 1 failed, 33 passed in 1263.81s (0:21:03) ===================

davydov-d · 2022-11-29T10:32:50Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3573232976
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3573232976
🐛 https://gradle.com/s/smx6pl3qosddq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream avail...
================== 1 failed, 33 passed in 1331.13s (0:22:11) ===================

davydov-d · 2022-11-29T12:15:14Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3573970077
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3573970077
🐛 https://gradle.com/s/i2jdk4crcjbfu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream usage...
================== 1 failed, 33 passed in 1229.82s (0:20:29) ===================

davydov-d · 2022-11-29T13:27:42Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3574526039
✅ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3574526039
Python tests coverage:

Name                        Stmts   Miss  Cover
-----------------------------------------------
source_twilio/auth.py           8      0   100%
source_twilio/__init__.py       2      0   100%
source_twilio/source.py        28      2    93%
source_twilio/streams.py      279     21    92%
-----------------------------------------------
TOTAL                         317     23    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

All Passed

alafanechere

Approving changes made to airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml on behalf of the connector operations team.

alafanechere · 2022-11-29T15:05:43Z

airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml

+          - name: "alerts"
+            bypass_reason: "alerts are transient - they emerge and disappear from time to time"
+          - name: "dependent_phone_numbers"
+            bypass_reason: "stream not filled yet"
+          - name: "conference_participants"
+            bypass_reason: "stream not filled yet"
+          - name: "available_phone_numbers_local"
+            bypass_reason: "very volatile data"
+          - name: "available_phone_numbers_mobile"
+            bypass_reason: "very volatile data"
+          - name: "available_phone_numbers_toll_free"
+            bypass_reason: "very volatile data"
+          - name: "usage_records"
+            bypass_reason: "very volatile data"


I logged an issue to get better seeding on this account: https://github.com/airbytehq/airbyte/issues/19876

davydov-d · 2022-11-29T16:16:33Z

/publish connector=connectors/source-twilio

🕑 Publishing the following connectors:
connectors/source-twilio
https://github.com/airbytehq/airbyte/actions/runs/3575947189

Connector	Did it publish?	Were definitions generated?
connectors/source-twilio	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

#1046 source twilio: fix generating date ranges

4905755

octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/twilio labels Nov 16, 2022

#1046 source twilio - upd changelog

4f4db6d

davydov-d requested review from darynaishchenko, girarda, grubberr and pedroslopez November 16, 2022 14:20

girarda approved these changes Nov 16, 2022

View reviewed changes

#1046 source twilio: add freezegun to requirements

d6b8bb8

davydov-d added 2 commits November 21, 2022 11:49

Merge branch 'master' into ddavydov/#1046-source-twilio-fix-date-ranges

8eaed85

#1046 source twilio: fix incremental tests

0462c6c

#1046 source twilio: fix expected records

5376ee6

#1046 source twilio - fix expected records

df2782e

Merge branch 'master' into ddavydov/#1046-source-twilio-fix-date-ranges

a94bba3

Merge branch 'master' into ddavydov/#1046-source-twilio-fix-date-ranges

8ef9f51

#1046 source twilio: cut expected records

9dc0860

#1046 source twilio: cut expected records

49149ef

alafanechere approved these changes Nov 29, 2022

View reviewed changes

Merge branch 'master' into ddavydov/#1046-source-twilio-fix-date-ranges

5ce21bc

auto-bump connector version

7bbfc04

octavia-squidington-iii temporarily deployed to more-secrets November 29, 2022 17:04 Inactive

davydov-d merged commit 34c0e95 into master Nov 29, 2022

davydov-d deleted the ddavydov/#1046-source-twilio-fix-date-ranges branch November 29, 2022 17:29

This was referenced Nov 29, 2022

Source Twilio: enable high test strictness level in SAT #19069

Closed

Source Twilio: enable high test strictness level in SAT #19182

Closed

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

davydov-d mentioned this pull request Jan 16, 2023

Source Twilio: enable high test strictness level in SAT #21336

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source twilio: fix generating date ranges #19479

Source twilio: fix generating date ranges #19479

davydov-d commented Nov 16, 2022

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 23, 2022

davydov-d commented Nov 23, 2022

davydov-d commented Nov 23, 2022

davydov-d commented Nov 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

alafanechere Nov 29, 2022

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

Source twilio: fix generating date ranges #19479

Source twilio: fix generating date ranges #19479

Conversation

davydov-d commented Nov 16, 2022

What

How

davydov-d commented Nov 16, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 16, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 21, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 21, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Nov 23, 2022

davydov-d commented Nov 23, 2022

davydov-d commented Nov 23, 2022

davydov-d commented Nov 28, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere Nov 29, 2022

Choose a reason for hiding this comment

davydov-d commented Nov 29, 2022 • edited by github-actions bot Loading

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 16, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 21, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 28, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading

davydov-d commented Nov 29, 2022 •

edited by github-actions bot

Loading