🐛 Source Marketo: improve field conversion to conform schema #8483

grubberr · 2021-12-03T11:33:43Z

Signed-off-by: Sergey Chvalyuk [email protected]

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

def normalize_datetime - I propose make date-time normalization more explicit than just trim chars.
MarketoExportBase.parse_response - improved to be more robust for cases like in (3).

3 . For new stream activities_new_lead we get different field names on discover and read steps WE NEED TO FIX IT!!!
Tthis happened because discover and read return different names ( which probaly need to be normalized to the same name)

  'API Method Name' -> 'a_p_i_method_name'
  'api method name' -> 'apimethodname'
  
  'Request Id' -> 'request_id'
  "request id" -> 'requestid'

  'Modifying User' -> 'modifying_user'
  'modifying user' -> 'modifyinguser'
  
  WE NEED TO FIX IT !!!!

Current implementation of utils.clean_string pretty 'magic' and can give ugly-looking results:

'Step ID' -> 'step_i_d'
"Campaign Run ID" -> 'campaign_run_i_d'
"SFDC Type" -> "s_f_d_c_type"
"Webpage ID" -> "webpage_i_d"

but we can live with it

I think we can to get rid of utils.clean_string and just use s.lower().replace(' ', '_')

Orig Value                         utils.clean_string            s.lower().replace(' ', '_')
-----------------------------------------------------------------------------------------------
Add to List                      addto_list                      add_to_list
Add to Nurture                   addto_nurture                   add_to_nurture
Add to Opportunity               addto_opportunity               add_to_opportunity
Change Status in Progression     change_statusin_progression     change_status_in_progression
Merged in Sales                  mergedin_sales                  merged_in_sales
Push Lead to Marketo             push_leadto_marketo             push_lead_to_marketo
Received by                      receivedby                      received_by
Received Forward to Friend Email received_forwardto_friend_email received_forward_to_friend_email
Remove from List                 removefrom_list                 remove_from_list
Remove from Opportunity          removefrom_opportunity          remove_from_opportunity
Reply to Sales Email             replyto_sales_email             reply_to_sales_email
Sent by                          sentby                          sent_by
Sent Forward to Friend Email     sent_forwardto_friend_email     sent_forward_to_friend_email

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2021-12-04T20:29:39Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1539667649
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1539667649
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 source_marketo/__init__.py       2      0   100%
	 source_marketo/source.py       254    165    35%
	 source_marketo/utils.py         33      8    76%
	 ------------------------------------------------
	 TOTAL                          289    173    40%

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2021-12-08T07:38:20Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1552994430
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1552994430
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 source_marketo/__init__.py       2      0   100%
	 source_marketo/source.py       250    161    36%
	 source_marketo/utils.py         33      8    76%
	 ------------------------------------------------
	 TOTAL                          285    169    41%

sherifnada · 2021-12-10T17:13:39Z

airbyte-integrations/connectors/source-marketo/source_marketo/utils.py

@@ -37,8 +37,15 @@ def clean_string(string: str) -> str:
    "updatedat" -> "updatedat"
    """

-    abbreviations = ("URL", "GUID", "IP")
-    if any(map(lambda w: w in string, abbreviations)):
+    fix = {


can we have unit tests?

I have added more cases for this unit_test 'test_clean_string'

Signed-off-by: Sergey Chvalyuk <[email protected]>

antixar

LGTM!
Please publish your changes

grubberr · 2021-12-15T10:07:29Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582183707
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582183707
🐛

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582183707
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582183707
🐛

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2021-12-15T10:29:52Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582266662
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582266662
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 source_marketo/__init__.py       2      0   100%
	 source_marketo/source.py       250    161    36%
	 source_marketo/utils.py         33      8    76%
	 ------------------------------------------------
	 TOTAL                          285    169    41%

grubberr · 2021-12-15T12:03:30Z

/publish connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582602615
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582602615

This reverts commit 52fc276.

Signed-off-by: Sergey Chvalyuk <[email protected]>

…hq#8483) * normalize_datetime added, better parse_response * clean_string updated * get_json_schema updated * bump version 0.1.1 -> 0.1.2 Signed-off-by: Sergey Chvalyuk <[email protected]>

normalize_datetime added, better parse_response

984ee5a

Signed-off-by: Sergey Chvalyuk <[email protected]>

github-actions bot added the area/connectors Connector related issues label Dec 3, 2021

bump version 0.1.0 -> 0.1.1

bc363d3

Signed-off-by: Sergey Chvalyuk <[email protected]>

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 3, 2021

grubberr temporarily deployed to more-secrets December 3, 2021 11:40 Inactive

grubberr added 2 commits December 3, 2021 16:27

Merge branch 'master' into grubberr/7776-source-marketo

5cac198

improve normalize_datetime

c79fe9e

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr self-assigned this Dec 3, 2021

grubberr temporarily deployed to more-secrets December 3, 2021 14:32 Inactive

grubberr requested review from antixar and lazebnyi December 3, 2021 14:53

lazebnyi linked an issue Dec 3, 2021 that may be closed by this pull request

Fix BigQuery destination normalization to accept any date-time format. #7776

Closed

grubberr added 2 commits December 4, 2021 22:02

Merge branch 'master' into grubberr/7776-source-marketo

34e6515

clean_string updated

2ee9af2

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 4, 2021 20:05 Inactive

clean_string updated (2)

2ed3801

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 4, 2021 20:11 Inactive

jrhizor temporarily deployed to more-secrets December 4, 2021 20:31 Inactive

Merge branch 'master' into grubberr/7776-source-marketo

9baedcf

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 7, 2021 11:54 Inactive

get_json_schema updated

e5c89e9

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 8, 2021 07:11 Inactive

grubberr added 2 commits December 8, 2021 09:25

Merge branch 'master' into grubberr/7776-source-marketo

612741f

add comment to normalize_datetime

e068e4e

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 8, 2021 07:33 Inactive

bump version 0.1.1 -> 0.1.2

23fd3a8

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 8, 2021 07:35 Inactive

jrhizor temporarily deployed to more-secrets December 8, 2021 07:40 Inactive

grubberr temporarily deployed to more-secrets December 9, 2021 12:04 Inactive

grubberr requested a review from lazebnyi December 9, 2021 12:23

lazebnyi approved these changes Dec 10, 2021

View reviewed changes

lazebnyi mentioned this pull request Dec 10, 2021

Marketo source fails during sync #7286

Closed

sherifnada reviewed Dec 10, 2021

View reviewed changes

grubberr added 2 commits December 13, 2021 10:53

Merge branch 'master' into grubberr/7776-source-marketo

4bbec28

unit_test.py updated

16607fa

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 13, 2021 09:05 Inactive

grubberr requested a review from sherifnada December 13, 2021 09:07

antixar approved these changes Dec 15, 2021

View reviewed changes

Merge branch 'master' into grubberr/7776-source-marketo

997598b

grubberr temporarily deployed to more-secrets December 15, 2021 10:05 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 10:09 Inactive

github secret -> GSM

52fc276

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 15, 2021 10:23 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 10:31 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 12:05 Inactive

grubberr added 3 commits December 15, 2021 16:58

Revert "github secret -> GSM"

28a25ef

This reverts commit 52fc276.

Merge branch 'master' into grubberr/7776-source-marketo

37dcc83

source_specs.yaml updated

e23d287

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 15, 2021 15:04 Inactive

grubberr merged commit b7b7886 into master Dec 15, 2021

grubberr deleted the grubberr/7776-source-marketo branch December 15, 2021 15:06

jrhizor mentioned this pull request Dec 16, 2021

Bump Airbyte version from 0.33.12-alpha to 0.34.0-alpha #8834

Merged

lazebnyi mentioned this pull request Jan 17, 2022

🐛 Source Marketo: Fix window in days validation issue #7594

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Marketo: improve field conversion to conform schema #8483

🐛 Source Marketo: improve field conversion to conform schema #8483

grubberr commented Dec 3, 2021 •

edited

Loading

grubberr commented Dec 4, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 8, 2021 •

edited by github-actions bot

Loading

sherifnada Dec 10, 2021

grubberr Dec 13, 2021

antixar left a comment

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading

🐛 Source Marketo: improve field conversion to conform schema #8483

🐛 Source Marketo: improve field conversion to conform schema #8483

Conversation

grubberr commented Dec 3, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

grubberr commented Dec 4, 2021 • edited by github-actions bot Loading

grubberr commented Dec 8, 2021 • edited by github-actions bot Loading

sherifnada Dec 10, 2021

Choose a reason for hiding this comment

grubberr Dec 13, 2021

Choose a reason for hiding this comment

antixar left a comment

Choose a reason for hiding this comment

grubberr commented Dec 15, 2021 • edited by github-actions bot Loading

grubberr commented Dec 15, 2021 • edited by github-actions bot Loading

grubberr commented Dec 15, 2021 • edited by github-actions bot Loading

grubberr commented Dec 3, 2021 •

edited

Loading

grubberr commented Dec 4, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 8, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading

grubberr commented Dec 15, 2021 •

edited by github-actions bot

Loading