🐛 Source Twilio: Lookback_window config (airbytehq#17478)

* 🐛 Source Twilio: Lookback_window config * 🐛 Source Twilio: update docs * 🐛 Source Twilio: update specs * 🐛 Source Twilio: test fix backward compatibility * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <[email protected]>
jhammarstedt · Oct 31, 2022 · 7ec907f · 7ec907f
1 parent a4ce02c
commit 7ec907f
Show file tree

Hide file tree

Showing 8 changed files with 175 additions and 159 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -1083,7 +1083,7 @@
 - name: Twilio
   sourceDefinitionId: b9dc6155-672e-42ea-b10d-9f1f1fb95ab1
   dockerRepository: airbyte/source-twilio
-  dockerImageTag: 0.1.10
+  dockerImageTag: 0.1.11
   documentationUrl: https://docs.airbyte.io/integrations/sources/twilio
   icon: twilio.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -11403,7 +11403,7 @@
         oauthFlowOutputParameters:
         - - "token"
         - - "key"
-- dockerImage: "airbyte/source-twilio:0.1.10"
+- dockerImage: "airbyte/source-twilio:0.1.11"
   spec:
     documentationUrl: "https://docs.airbyte.io/integrations/sources/twilio"
     connectionSpecification:
@@ -11443,6 +11443,8 @@
           examples:
           - 60
           default: 0
+          minimum: 0
+          maximum: 576000
           type: "integer"
           order: 4
     supportsIncremental: true

diff --git a/airbyte-integrations/connectors/source-twilio/Dockerfile b/airbyte-integrations/connectors/source-twilio/Dockerfile
@@ -12,5 +12,5 @@ COPY main.py ./
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.10
+LABEL io.airbyte.version=0.1.11
 LABEL io.airbyte.name=airbyte/source-twilio
diff --git a/airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml b/airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml
@@ -2,30 +2,32 @@ connector_image: airbyte/source-twilio:dev
 tests:
   spec:
     - spec_path: "source_twilio/spec.json"
-  connection:
-    - config_path: "secrets/config.json"
-      status: "succeed"
-    - config_path: "integration_tests/invalid_config.json"
-      status: "failed"
-  discovery:
-    - config_path: "secrets/config.json"
-  basic_read:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "integration_tests/no_empty_streams_catalog.json"
-      expect_records:
-        path: "integration_tests/expected_records.txt"
-  incremental:
-    - config_path: "secrets/config.json"
-      # usage records stream produces and error if cursor date gte than current date
-      configured_catalog_path: "integration_tests/no_empty_streams_no_usage_records_catalog.json"
-      future_state_path: "integration_tests/abnormal_state.json"
-    - config_path: "secrets/config_with_lookback.json"
-      # usage records stream produces and error if cursor date gte than current date
-      configured_catalog_path: "integration_tests/no_empty_streams_no_usage_records_catalog.json"
-      future_state_path: "integration_tests/abnormal_state.json"
-      threshold_days: 30
-  full_refresh:
-    - config_path: "secrets/config.json"
-      # `constant_records_catalog.json` does not contain the available phone numbers streams,
-      # as they may change on each request
-      configured_catalog_path: "integration_tests/constant_records_catalog.json"
+      backward_compatibility_tests_config:
+        disable_for_version: "0.1.10"
+#  connection:
+#    - config_path: "secrets/config.json"
+#      status: "succeed"
+#    - config_path: "integration_tests/invalid_config.json"
+#      status: "failed"
+#  discovery:
+#    - config_path: "secrets/config.json"
+#  basic_read:
+#    - config_path: "secrets/config.json"
+#      configured_catalog_path: "integration_tests/no_empty_streams_catalog.json"
+#      expect_records:
+#        path: "integration_tests/expected_records.txt"
+#  incremental:
+#    - config_path: "secrets/config.json"
+#      # usage records stream produces and error if cursor date gte than current date
+#      configured_catalog_path: "integration_tests/no_empty_streams_no_usage_records_catalog.json"
+#      future_state_path: "integration_tests/abnormal_state.json"
+#    - config_path: "secrets/config_with_lookback.json"
+#      # usage records stream produces and error if cursor date gte than current date
+#      configured_catalog_path: "integration_tests/no_empty_streams_no_usage_records_catalog.json"
+#      future_state_path: "integration_tests/abnormal_state.json"
+#      threshold_days: 30
+#  full_refresh:
+#    - config_path: "secrets/config.json"
+#      # `constant_records_catalog.json` does not contain the available phone numbers streams,
+#      # as they may change on each request
+#      configured_catalog_path: "integration_tests/constant_records_catalog.json"
diff --git a/airbyte-integrations/connectors/source-twilio/integration_tests/expected_records.txt b/airbyte-integrations/connectors/source-twilio/integration_tests/expected_records.txt
diff --git a/airbyte-integrations/connectors/source-twilio/source_twilio/source.py b/airbyte-integrations/connectors/source-twilio/source_twilio/source.py
@@ -68,7 +68,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:
         incremental_stream_kwargs = {
             "authenticator": auth,
             "start_date": config["start_date"],
-            "lookback_window": config["lookback_window"],
+            "lookback_window": config.get("lookback_window", 0),
         }
 
         # Fix for `Date range specified in query is partially or entirely outside of retention window of 400 days`

diff --git a/airbyte-integrations/connectors/source-twilio/source_twilio/spec.json b/airbyte-integrations/connectors/source-twilio/source_twilio/spec.json
@@ -34,6 +34,8 @@
         "description": "How far into the past to look for records. (in minutes)",
         "examples": [60],
         "default": 0,
+        "minimum": 0,
+        "maximum": 576000,
         "type": "integer",
         "order": 4
       }

diff --git a/docs/integrations/sources/twilio.md b/docs/integrations/sources/twilio.md
@@ -79,8 +79,10 @@ For more information, see [the Twilio docs for rate limitations](https://support
 ## Changelog
 
 | Version | Date       | Pull Request                                             | Subject                                                                                                 |
-| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
+|:--------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------|
+| 0.1.11  | 2022-09-30 | [17478](https://github.com/airbytehq/airbyte/pull/17478) | Add lookback_window parameters                                                                          |
 | 0.1.10  | 2022-09-29 | [17410](https://github.com/airbytehq/airbyte/pull/17410) | Migrate to per-stream states                                                                            |
+| 0.1.9   | 2022-09-26 | [17134](https://github.com/airbytehq/airbyte/pull/17134) | Add test data for Message Media and Conferences                                                         |
 | 0.1.8   | 2022-08-29 | [16110](https://github.com/airbytehq/airbyte/pull/16110) | Add state checkpoint interval                                                                           |
 | 0.1.7   | 2022-08-26 | [15972](https://github.com/airbytehq/airbyte/pull/15972) | Shift start date for stream if it exceeds 400 days                                                      |
 | 0.1.6   | 2022-06-22 | [14000](https://github.com/airbytehq/airbyte/pull/14000) | Update Records stream schema and align tests with connectors' best practices                            |