Source google-analytics-v4: fix pagination (airbytehq#17943)

* airbytehq#787 oncall - source google-analytics: fix pagination * source google-analytics-v4: upd changelog * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <[email protected]>
jhammarstedt · Oct 31, 2022 · 445737a · 445737a
1 parent ea91909
commit 445737a
Show file tree

Hide file tree

Showing 9 changed files with 28 additions and 12 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -386,7 +386,7 @@
 - name: Google Analytics (Universal Analytics)
   sourceDefinitionId: eff3616a-f9c3-11eb-9a03-0242ac130003
   dockerRepository: airbyte/source-google-analytics-v4
-  dockerImageTag: 0.1.29
+  dockerImageTag: 0.1.30
   documentationUrl: https://docs.airbyte.com/integrations/sources/google-analytics-universal-analytics
   icon: google-analytics.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -3865,7 +3865,7 @@
         oauthFlowOutputParameters:
         - - "access_token"
         - - "refresh_token"
-- dockerImage: "airbyte/source-google-analytics-v4:0.1.29"
+- dockerImage: "airbyte/source-google-analytics-v4:0.1.30"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/google-analytics-universal-analytics"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-google-analytics-v4/Dockerfile b/airbyte-integrations/connectors/source-google-analytics-v4/Dockerfile
@@ -12,5 +12,5 @@ COPY main.py ./
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.29
+LABEL io.airbyte.version=0.1.30
 LABEL io.airbyte.name=airbyte/source-google-analytics-v4
diff --git a/...e-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py b/...e-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py
@@ -132,9 +132,12 @@ def path(self, **kwargs: Any) -> str:
         return "./reports:batchGet"
 
     def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
-        next_page = response.json().get("nextPageToken")
-        if next_page:
-            return {"pageToken": next_page}
+        reports = response.json().get(self.report_field, [])
+        for report in reports:
+            # since we're requesting just one report at a time, the first report in the response is enough
+            next_page = report.get("nextPageToken")
+            if next_page:
+                return {"pageToken": next_page}
 
     def should_retry(self, response: requests.Response) -> bool:
         """

diff --git a/...e-integrations/connectors/source-google-analytics-v4/unit_tests/response_golden_data.json b/...e-integrations/connectors/source-google-analytics-v4/unit_tests/response_golden_data.json
@@ -40,8 +40,7 @@
             "values": ["3"]
           }
         ]
-      },
-      "nextPageToken": "1"
+      }
     }
   ]
 }
diff --git a/...tegrations/connectors/source-google-analytics-v4/unit_tests/response_non_golden_data.json b/...tegrations/connectors/source-google-analytics-v4/unit_tests/response_non_golden_data.json
@@ -39,8 +39,7 @@
             "values": ["3"]
           }
         ]
-      },
-      "nextPageToken": "1"
+      }
     }
   ]
 }
diff --git a/...integrations/connectors/source-google-analytics-v4/unit_tests/response_with_sampling.json b/...integrations/connectors/source-google-analytics-v4/unit_tests/response_with_sampling.json
@@ -41,8 +41,7 @@
             "values": ["3"]
           }
         ]
-      },
-      "nextPageToken": "1"
+      }
     }
   ]
 }
diff --git a/airbyte-integrations/connectors/source-google-analytics-v4/unit_tests/unit_test.py b/airbyte-integrations/connectors/source-google-analytics-v4/unit_tests/unit_test.py
@@ -397,3 +397,18 @@ def test_is_data_golden_flag_missing_equals_false(
     for message in source.read(logging.getLogger(), test_config, configured_catalog):
         if message.type == Type.RECORD:
             assert message.record.data["isDataGolden"] is False
+
+
+@pytest.mark.parametrize(
+    "configured_response, expected_token",
+    (
+        ({}, None),
+        ({"reports": []}, None),
+        ({"reports": [{"data": {}, "columnHeader": {}}]}, None),
+        ({"reports": [{"data": {}, "columnHeader": {}, "nextPageToken": 100000}]}, {"pageToken": 100000}),
+    ),
+)
+def test_next_page_token(test_config, configured_response, expected_token):
+    response = MagicMock(json=MagicMock(return_value=configured_response))
+    token = GoogleAnalyticsV4Stream(test_config).next_page_token(response)
+    assert token == expected_token
diff --git a/docs/integrations/sources/google-analytics-universal-analytics.md b/docs/integrations/sources/google-analytics-universal-analytics.md
@@ -155,6 +155,7 @@ Incremental sync is supported only if you add `ga:date` dimension to your custom
 
 | Version | Date       | Pull Request                                             | Subject                                                                                      |
 |:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------|
+| 0.1.30  | 2022-10-13 | [17943](https://github.com/airbytehq/airbyte/pull/17943) | Fix pagination                                                                               |
 | 0.1.29  | 2022-10-12 | [17905](https://github.com/airbytehq/airbyte/pull/17905) | Handle exceeded daily quota gracefully                                                       |
 | 0.1.28  | 2022-09-24 | [16920](https://github.com/airbytehq/airbyte/pull/16920) | Added segments and filters to custom reports                                                 |
 | 0.1.27  | 2022-10-07 | [17717](https://github.com/airbytehq/airbyte/pull/17717) | Improve CHECK by using `ga:hits` metric.                                                     |