🎉 Google Analytics connector is failing #3648

po3na4skld · 2021-05-26T21:01:34Z

What

It solves #3355.

How

Added filter into the GoogleAnalyticsSingerSource._get_reports_file_path so that it could remove non used in catalog provided by user. Also removed duplicated "title" key in spec.json file.

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

source.py
spec.json

yevhenii-ldv

LGTM

sherifnada · 2021-05-27T18:02:48Z

...ntegrations/connectors/source-googleanalytics-singer/source_googleanalytics_singer/source.py

@@ -59,19 +61,26 @@ def _validate_custom_reports(self, custom_reports_data: List[dict]):
                error_messages.append(error.message)
            raise Exception("An error occurred during custom_reports data validation: " + "; ".join(error_messages))

+    def read_catalog(self, catalog_path: str) -> ConfiguredAirbyteCatalog:


this method is not called anywhere?

It is called when performing read operation, I redefined this function from a base class to save and then filter out streams we want to use

@po3na4skld, This method is called during the Read method, and when the Discover method is launched, no one stream will found, so we will get zero streams and we will not be able to read the data.

@yevhenii-ldv for now it works as before

yevhenii-ldv

See above in comments

yevhenii-ldv · 2021-06-02T10:50:10Z

...ntegrations/connectors/source-googleanalytics-singer/source_googleanalytics_singer/source.py

    def _get_reports_file_path(self, temp_dir: str, custom_reports_data: List[dict]) -> str:
        report_definition = (
            json.loads(pkgutil.get_data("tap_google_analytics", "defaults/default_report_definition.json")) + custom_reports_data
        )
+        if self.reports_to_read is not None:


if self.reports_to_read

yevhenii-ldv

LGTM, one small comment

keu · 2021-06-03T10:16:25Z

@po3na4skld I see some strange files in PR that don't belong here, could you please merge with master and ensure there are no extra files.

po3na4skld · 2021-06-03T10:18:04Z

@po3na4skld I see some strange files in PR that don't belong here, could you please merge with master and ensure there are no extra files.

I think it is because I did chmod 777 on airbyte folder and after gradlew format they were added to the commits @keu you can see that there are not actual changes to these files. Merged master in this branch

keu

please use this snippet to rollback changes
https://gist.github.com/liitfr/86c77bf0ea4a627ab9b62ad30c32c251

po3na4skld · 2021-06-08T15:09:31Z

@davinchia
I added tests, but couldn't run them as the API tells that Daily Request Limit is reached and throws 429 HTTP error. The same when I run the read operation. I suppose I need to move this into blocked

davinchia · 2021-06-09T06:19:01Z

thanks @po3na4skld

@sherifnada is this related to #3958? Have we tried getting our limit increased?

po3na4skld · 2021-06-09T09:10:57Z

thanks @po3na4skld

@sherifnada is this related to #3958? Have we tried getting our limit increased?

@davinchia As I know for now we disabled any GA testing when merging to master in order to keep rate limit on an acceptable level for testing this PR

davinchia

@po3na4skld thanks for adding the test!

since we can't test this; I think it's fine to merge if you feel confident about it. we should make sure we eventually come back and turn it the tests on so we have confidence things are working well.

keu · 2021-06-11T19:24:36Z

/test connector=source-googleanalytics-singer

🕑 source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/929608854
❌ source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/929608854

airbyte-integrations/connectors/source-googleanalytics-singer/build.gradle

po3na4skld · 2021-06-15T08:52:36Z

/publish connector=connectors/source-googleanalytics-singer

🕑 connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/938672052
❌ connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/938672052

po3na4skld · 2021-06-15T10:41:28Z

/test connector=connectors/source-googleanalytics-singer

🕑 connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/938979005
✅ connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/938979005

yevhenii-ldv · 2021-06-15T13:20:19Z

/publish connector=connectors/source-googleanalytics-singer

🕑 connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/939421703
✅ connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/939421703

He passed it to another person: davinchia

added filter of streams and deleted duplicated title in spec.json

0043874

po3na4skld added this to the Connectors May 28th, 2021 milestone May 26, 2021

po3na4skld requested review from keu, yevhenii-ldv and vitaliizazmic May 26, 2021 21:01

po3na4skld self-assigned this May 26, 2021

auto-assign bot requested review from ChristopheDuong and masonwheeler May 26, 2021 21:01

po3na4skld linked an issue May 26, 2021 that may be closed by this pull request

Google Analytics Connector is failing #3355

Closed

yevhenii-ldv approved these changes May 27, 2021

View reviewed changes

sherifnada previously requested changes May 27, 2021

View reviewed changes

yevhenii-ldv suggested changes May 28, 2021

View reviewed changes

fixed discover returns empty streams

21b52b0

po3na4skld requested a review from yevhenii-ldv May 28, 2021 10:50

sherifnada modified the milestones: Connectors May 28th, 2021, Connectors June 11, 2021 May 28, 2021

yevhenii-ldv reviewed Jun 2, 2021

View reviewed changes

yevhenii-ldv approved these changes Jun 2, 2021

View reviewed changes

yevhenii-ldv removed request for masonwheeler, ChristopheDuong and keu June 2, 2021 10:50

po3na4skld requested a review from keu June 2, 2021 11:55

pr changes

e81a465

pr changes

4d096bc

keu suggested changes Jun 3, 2021

View reviewed changes

po3na4skld added 3 commits June 3, 2021 17:05

chmod on files changed

c5321ce

chmod on files changed

d2755bd

chmod on files changed

5d5ca42

added acceptance tests

6fed47d

added pre commit config

5253387

po3na4skld added 2 commits June 9, 2021 21:43

fixed chmod on amazon and facebook files

e77c90b

fixed chmod on amazon and facebook files

f5ae757

davinchia approved these changes Jun 10, 2021

View reviewed changes

VasylLazebnyk removed this from the Connectors June 11, 2021 milestone Jun 11, 2021

keu reviewed Jun 11, 2021

View reviewed changes

airbyte-integrations/connectors/source-googleanalytics-singer/build.gradle Show resolved Hide resolved

version bump

ba92bc6

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jun 15, 2021

po3na4skld and others added 3 commits June 15, 2021 10:16

version bump

668fd75

Merge branch 'master' into KC-Google-Analytics-Connector-is-failing

0a13233

build file fix

13f6345

po3na4skld added 2 commits June 15, 2021 12:35

removed unrelated files

43a2be5

removed unrelated files

12b5225

po3na4skld added 4 commits June 15, 2021 15:06

removed unrelated files

83c7e76

removed unrelated files

6095166

removed unrelated files

1de9852

removed unrelated files

d1775f1

po3na4skld merged commit 856abc3 into master Jun 15, 2021

po3na4skld deleted the KC-Google-Analytics-Connector-is-failing branch June 15, 2021 17:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Google Analytics connector is failing #3648

🎉 Google Analytics connector is failing #3648

po3na4skld commented May 26, 2021 •

edited

Loading

yevhenii-ldv left a comment

sherifnada May 27, 2021

po3na4skld May 28, 2021

yevhenii-ldv May 28, 2021

po3na4skld May 28, 2021

po3na4skld May 28, 2021

yevhenii-ldv left a comment

yevhenii-ldv Jun 2, 2021

yevhenii-ldv left a comment

keu commented Jun 3, 2021

po3na4skld commented Jun 3, 2021 •

edited

Loading

keu left a comment •

edited

Loading

po3na4skld commented Jun 8, 2021

davinchia commented Jun 9, 2021

po3na4skld commented Jun 9, 2021

davinchia left a comment •

edited

Loading

keu commented Jun 11, 2021 •

edited by github-actions bot

Loading

po3na4skld commented Jun 15, 2021 •

edited by github-actions bot

Loading

po3na4skld commented Jun 15, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Jun 15, 2021 •

edited by github-actions bot

Loading

🎉 Google Analytics connector is failing #3648

🎉 Google Analytics connector is failing #3648

Conversation

po3na4skld commented May 26, 2021 • edited Loading

What

How

Pre-merge Checklist

Recommended reading order

yevhenii-ldv left a comment

Choose a reason for hiding this comment

sherifnada May 27, 2021

Choose a reason for hiding this comment

po3na4skld May 28, 2021

Choose a reason for hiding this comment

yevhenii-ldv May 28, 2021

Choose a reason for hiding this comment

po3na4skld May 28, 2021

Choose a reason for hiding this comment

po3na4skld May 28, 2021

Choose a reason for hiding this comment

yevhenii-ldv left a comment

Choose a reason for hiding this comment

yevhenii-ldv Jun 2, 2021

Choose a reason for hiding this comment

yevhenii-ldv left a comment

Choose a reason for hiding this comment

keu commented Jun 3, 2021

po3na4skld commented Jun 3, 2021 • edited Loading

keu left a comment • edited Loading

Choose a reason for hiding this comment

po3na4skld commented Jun 8, 2021

davinchia commented Jun 9, 2021

po3na4skld commented Jun 9, 2021

davinchia left a comment • edited Loading

Choose a reason for hiding this comment

keu commented Jun 11, 2021 • edited by github-actions bot Loading

po3na4skld commented Jun 15, 2021 • edited by github-actions bot Loading

po3na4skld commented Jun 15, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented Jun 15, 2021 • edited by github-actions bot Loading

po3na4skld commented May 26, 2021 •

edited

Loading

po3na4skld commented Jun 3, 2021 •

edited

Loading

keu left a comment •

edited

Loading

davinchia left a comment •

edited

Loading

keu commented Jun 11, 2021 •

edited by github-actions bot

Loading

po3na4skld commented Jun 15, 2021 •

edited by github-actions bot

Loading

po3na4skld commented Jun 15, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented Jun 15, 2021 •

edited by github-actions bot

Loading