🔧 Pinterest Schema updated #9537

rfortmann-ewolff · 2022-01-16T20:24:58Z

What

ad_id and account_id are non-existent in the destination (BigQuery). The schema has a typo for ad_id and account_id.

How

The schemas have been updated.

CLAassistant · 2022-01-16T20:25:03Z

All committers have signed the CLA.

* add mailchimp oauth support * add PR * fix creds * upd spec * format * upd creds * upd auth for different creds * rename creds * rename creds * change ref in campaigns.json * upd timeout_seconds * merge * add oauth java part * add java test * bump version * update spec * add anotation * upd spec * upd spec * upd * upd tests * format * upd * upd * add state * add invalid_config * bump version * format

…reams to fix error with `502` HTTP status code (#9492) * Source Github: Remove optional parameter Accept for reaction's streams to fix error with 502 HTTP status code

* change list of scopes * upd changelog * bump version

marcosmarxm · 2022-01-18T03:25:48Z

@harshithmullapudi please review this PR.

* fixed test which check incorrect cred config * Added oauth2 authentication * Added oauth creds * fixed formatting * added oauth2 spec section, added missing type hints * Added java part of Square OAuth * fixed checkstyle * removed commented code * added support for old format of spec.json files, updated change logs docs * renamed spec property 'authentication' to default 'credentials'. fixed changes in java part * recovered empty files * updated OAuthImplementationFactory.java * fixed issue with autheticator for sub streams, added config catalog with all streams, updated docs * use advanced_auth * added advanced_auth * moved scopes to private property * updated source version * Revert "updated source version" This reverts commit ce3d061. * updated source version * added new version for airbyte index Co-authored-by: ievgeniit <[email protected]>

...-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_account_analytics.json

airbyte-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_analytics.json

...te-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_group_analytics.json

...te-integrations/connectors/source-pinterest/source_pinterest/schemas/campaign_analytics.json

harshithmullapudi · 2022-01-18T08:19:34Z

Hey @rfortmann-ewolff https://docs.airbyte.com/connector-development#updating-an-existing-connector can you check this

…8383) * add Dataset ID parse method * add BigQuery Destination unit test * update change log * fit to the latest code base * update change log * change var name to const name * change public method to private * add test cases for testGetDatasetIdFail * add integration test for dataset-id prefixed with project-id * fix getDatasetId * add comment to parameterized test provider * update docker image versions * update docker image versions again

rfortmann-ewolff · 2022-01-18T12:58:52Z

Hey @rfortmann-ewolff https://docs.airbyte.com/connector-development#updating-an-existing-connector can you check this

all tests pass:

no new tests required
no documentation update required

* fix next_page_token * fix BULK API * fix BUlk incremental stream * added unit test and comments * format code * bump version * updated spec and def yaml Co-authored-by: auganbay <[email protected]>

* New source: Chartmogul * update changelog * bootstrap.md * reorder schema * make acceptance test pass * add chartmogul to builds.md * Update airbyte-integrations/connectors/source-chartmogul/source_chartmogul/source.py Co-authored-by: Augustin <[email protected]> * Update airbyte-integrations/connectors/source-chartmogul/source_chartmogul/source.py Co-authored-by: Augustin <[email protected]> * address comments * fix unused imports * update source_specs Co-authored-by: alafanechere <[email protected]> Co-authored-by: Augustin <[email protected]>

* update desrcription for bq destination setup * bump version

* use BULK for the first sync, REST for incremental sync * if stream contains compound data or/and base64 use always REST * fix get stream state from connector state * fix integration test * refactor catalog name * format code * refactor unit tests * refactor unit tests 2 * format code 2 * Set additionalProperties to true not to break test temporarily * fix unit test and remove unnecessary filtering fields * bump version * updated spec and def yaml Co-authored-by: auganbay <[email protected]>

Signed-off-by: Sergey Chvalyuk <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]>

Co-authored-by: benmoriceau <[email protected]>

* Test sentry release step * Fix bug * Fix typo * Cut off prefix * Fix env variables * Add back steps * Fix syntax error * Use startsWith to replace IMAGE_TYPE * Update title * Bump version in seed

* Update doc * Use empty dsn when sentry is not enabled * Bump version in seed

* cleanup airbyte_configs table * fix test * fix test

rfortmann-ewolff · 2022-02-01T09:31:09Z

You can rebase it with latest master first then

Even after the rebase, I still get the same error.

rfortmann-ewolff · 2022-02-01T10:28:14Z

I only rebased on my fork and not on the origin master - my bad.

…m:rfortmann-ewolff/airbyte into pinterest-schema

rfortmann-ewolff · 2022-02-01T16:07:51Z

You can rebase it with latest master first then

configured_catalog = ConfiguredAirbyteCatalog(streams=[ConfiguredAirbyteStream(stream=AirbyteStream(name='boards', json_schema={'$schema': ...l: 'incremental'>, cursor_field=None, destination_sync_mode=<DestinationSyncMode.append: 'append'>, primary_key=None)])
allowed_empty_streams = set()

    def _validate_empty_streams(self, records, configured_catalog, allowed_empty_streams):
        """
        Only certain streams allowed to be empty
        """
        counter = Counter(record.stream for record in records)

        all_streams = set(stream.stream.name for stream in configured_catalog.streams)
        streams_with_records = set(counter.keys())
        streams_without_records = all_streams - streams_with_records

        streams_without_records = streams_without_records - allowed_empty_streams
>       assert not streams_without_records, f"All streams should return some records, streams without records: {streams_without_records}"
E       AssertionError: All streams should return some records, streams without records: {'ad_group_analytics', 'board_section_pins', 'boards', 'board_pins', 'ad_account_analytics', 'campaign_analytics', 'board_sections'}
E       assert not {'ad_account_analytics', 'ad_group_analytics', 'board_pins', 'board_section_pins', 'board_sections', 'boards', ...}

/usr/local/lib/python3.7/site-packages/source_acceptance_test/tests/test_core.py:289: AssertionError

I don't understand the error that is meant here.

harshithmullapudi · 2022-02-01T16:10:17Z

Hey looks like the rebase didn't happen the right way there are 176 commits I see can you look into that.

You can also try merging master into your branch

rfortmann-ewolff · 2022-02-01T16:56:27Z

Even though the schema change seemed easy and should not have produced much errors, I had to close my PR as tests were impossible to reproduce.

rfortmann-ewolff · 2022-02-01T17:02:28Z

If anyone wants to try to solve that schema change, here is where the typos take place:

airbyte/airbyte-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_account_analytics.json

Line 9 in 98b5aab

"AD_ACCOUNT_IDAD_ID": {

airbyte/airbyte-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_analytics.json

Line 9 in 98b5aab

"AD_ACCOUNT_IDAD_ID": {

airbyte/airbyte-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_group_analytics.json

Line 9 in 98b5aab

"AD_ACCOUNT_IDAD_ID": {

airbyte/airbyte-integrations/connectors/source-pinterest/source_pinterest/schemas/campaign_analytics.json

Line 9 in 98b5aab

"AD_ACCOUNT_IDAD_ID": {

🔧 Pinterest Schema updated

8071019

github-actions bot added the area/connectors Connector related issues label Jan 16, 2022

octavia-squidington-iii added the community label Jan 16, 2022

alafanechere and others added 6 commits January 17, 2022 09:27

🐙 octavia-cli: generate open api client (#9277)

d65b8f9

Source Hubspot: Some incremental CRM objects and engagements (#8887)

25fb7e7

🐛 Source Github: Remove optional parameter Accept for reaction's st…

cb6d9ab

…reams to fix error with `502` HTTP status code (#9492) * Source Github: Remove optional parameter Accept for reaction's streams to fix error with 502 HTTP status code

🎉 SurveyMonkey source: scopes change (#9508)

edc0925

* change list of scopes * upd changelog * bump version

Add section to UX handbook about pointing to page anchors

032b06d

marcosmarxm assigned harshithmullapudi Jan 18, 2022

harshithmullapudi suggested changes Jan 18, 2022

View reviewed changes

🎨 new line formatting

9535f40

rfortmann-ewolff mentioned this pull request Jan 18, 2022

[EPIC] Update schemas in source connector source-pinterest #9568

Closed

2 tasks

subodh1810 and others added 2 commits January 18, 2022 17:38

fix master formatting (#9571)

c4b365c

augan-rymkhan and others added 10 commits January 18, 2022 20:42

Source Salesforce: fix pagination in REST API streams (#9151)

62c433e

* fix next_page_token * fix BULK API * fix BUlk incremental stream * added unit test and comments * format code * bump version * updated spec and def yaml Co-authored-by: auganbay <[email protected]>

Destination BigQuery: update description for some fields (#9573)

0187efd

* update desrcription for bq destination setup * bump version

🐛 Fix syntax errors in our specification docs (#9576)

cbba9fa

🐛 Minor correction in the frontend tech-stack (#9547)

7a26246

add node strict check (#9335)

f3798ed

🐛 Prevent SVG icons from leaking into DOM (#9574)

38a5d84

Remove empty helm workflow (#9592)

cb8644e

Update fields in source-connectors specifications: google-ads (#9149)

a1e2b63

Signed-off-by: Sergey Chvalyuk <[email protected]> Co-authored-by: Serhii Chvaliuk <[email protected]>

octavia-squidington-iii and others added 4 commits January 31, 2022 16:15

Bump Airbyte version from 0.35.13-alpha to 0.35.14-alpha (#9944)

ff2f665

Co-authored-by: benmoriceau <[email protected]>

Add sentry release for connectors (#9940)

aa3e00d

* Test sentry release step * Fix bug * Fix typo * Cut off prefix * Fix env variables * Add back steps * Fix syntax error * Use startsWith to replace IMAGE_TYPE * Update title * Bump version in seed

🎉 Destination snowflake & bigquery: integrate with sentry (#9945)

8e8f402

* Update doc * Use empty dsn when sentry is not enabled * Bump version in seed

cleanup airbyte_configs table (#9437)

91fd734

* cleanup airbyte_configs table * fix test * fix test

tuliren and others added 6 commits February 1, 2022 01:44

🎉 Source e2e-test: integrate with Sentry (#9955)

a518247

🔧 Pinterest Schema updated

3d9c043

🎨 new line formatting

7efe322

🔖 Version Bump in Dockerfile

cc0461e

⬆️ Dependency in Airbyte connector index updated

5213921

🚧 Stashed changes

4826645

Merge branches 'pinterest-schema' and 'pinterest-schema' of github.co…

08ee3d1

…m:rfortmann-ewolff/airbyte into pinterest-schema

github-actions bot added area/api Related to the api area/documentation Improvements or additions to documentation area/frontend area/platform issues related to the platform area/protocol area/scheduler area/server area/worker Related to worker kubernetes normalization labels Feb 1, 2022

rfortmann-ewolff closed this Feb 1, 2022

rfortmann-ewolff deleted the pinterest-schema branch February 1, 2022 16:54

rfortmann-ewolff mentioned this pull request Feb 9, 2022

Pinterest Schema updated - 2nd try #10223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🔧 Pinterest Schema updated #9537

🔧 Pinterest Schema updated #9537

rfortmann-ewolff commented Jan 16, 2022 •

edited

Loading

CLAassistant commented Jan 16, 2022 •

edited

Loading

marcosmarxm commented Jan 18, 2022

harshithmullapudi commented Jan 18, 2022

rfortmann-ewolff commented Jan 18, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022 •

edited

Loading

harshithmullapudi commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

🔧 Pinterest Schema updated #9537

🔧 Pinterest Schema updated #9537

Conversation

rfortmann-ewolff commented Jan 16, 2022 • edited Loading

What

How

CLAassistant commented Jan 16, 2022 • edited Loading

marcosmarxm commented Jan 18, 2022

harshithmullapudi commented Jan 18, 2022

rfortmann-ewolff commented Jan 18, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022 • edited Loading

harshithmullapudi commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Feb 1, 2022

rfortmann-ewolff commented Jan 16, 2022 •

edited

Loading

CLAassistant commented Jan 16, 2022 •

edited

Loading

rfortmann-ewolff commented Feb 1, 2022 •

edited

Loading