🐛 Source SalesForce: better detect API type #16086

grubberr · 2022-08-29T12:03:37Z

Signed-off-by: Sergey Chvalyuk [email protected]

What

If stream has a lot of properties which overcome HTTP GET Request Size Limits 16384 bytes
try to use BULK API for such cases

How

Improve logic which detect which API to use REST API or BULK API

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-29T12:08:40Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2948387527
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2948387527
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/api.py               151     16    89%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     67    78%
source_salesforce/source.py             94     36    62%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  592    130    78%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/api.py               151     14    91%
source_salesforce/source.py             94      9    90%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     42    86%
--------------------------------------------------------
TOTAL                                  592     69    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-29T14:08:09Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2949179312

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-29T14:10:36Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2949196139
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2949196139
🐛 https://gradle.com/s/y6bheaeexry7e

Build Failed

Test summary info:

Could not find result summary

grubberr · 2022-08-29T18:00:16Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2950665277
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2950665277
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           307     73    76%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     36    62%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  592    142    76%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/api.py               151     14    91%
source_salesforce/source.py             94      9    90%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     42    86%
--------------------------------------------------------
TOTAL                                  592     69    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-30T09:28:27Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2955108263
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2955108263
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           307     67    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             94     36    62%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  592    136    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/api.py               151     14    91%
source_salesforce/source.py             94      9    90%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     42    86%
--------------------------------------------------------
TOTAL                                  592     69    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

edgao

couple questions for my understanding but otherwise this looks good

airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-30T16:25:59Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2957908324
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2957908324
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           307     67    78%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             91     35    62%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  589    135    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             91      8    91%
source_salesforce/api.py               151     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     42    86%
--------------------------------------------------------
TOTAL                                  589     68    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-08-30T18:47:50Z

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/2958741820

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grubberr · 2022-08-30T19:27:50Z

/publish connector=connectors/source-salesforce

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/2958985416

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grubberr · 2022-08-30T19:59:36Z

/publish connector=connectors/source-salesforce run-tests=false

grubberr · 2022-08-30T20:57:23Z

/publish connector=connectors/source-salesforce run-tests=false

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/2959512086

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grubberr · 2022-08-30T21:14:14Z

/publish connector=connectors/source-salesforce run-tests=false

🕑 Publishing the following connectors:
connectors/source-salesforce
https://github.com/airbytehq/airbyte/actions/runs/2959607679

Connector	Did it publish?	Were definitions generated?
connectors/source-salesforce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <[email protected]>

_get_api_type added

552e165

Signed-off-by: Sergey Chvalyuk <[email protected]>

github-actions bot added the area/connectors Connector related issues label Aug 29, 2022

octavia-squidington-iii added the connectors/source/salesforce label Aug 29, 2022

grubberr self-assigned this Aug 29, 2022

salesforce.md updated

1b686e9

Signed-off-by: Sergey Chvalyuk <[email protected]>

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 29, 2022

grubberr added 4 commits August 29, 2022 15:53

Merge branch 'master' into grubberr/oncall-412-source-salesforce-2

d4f92a2

MAX_HTTP_QUERY_STRING_SIZE -> MAX_URL_LENGTH

b4414b5

Signed-off-by: Sergey Chvalyuk <[email protected]>

reason: URI Too Long message added

08b7c2b

Signed-off-by: Sergey Chvalyuk <[email protected]>

MAX_URL_LENGTH -> REQUEST_SIZE_LIMITS

85d1787

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr requested review from bazarnov and roman-yermilov-gl August 29, 2022 14:08

MAX_URL_LENGTH -> REQUEST_SIZE_LIMITS (2)

9b39ced

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr requested review from Phlair and edgao August 29, 2022 22:09

grubberr added 2 commits August 30, 2022 12:27

Merge branch 'master' into grubberr/oncall-412-source-salesforce-2

a6694e8

Signed-off-by: Sergey Chvalyuk <[email protected]>

bump 0.1.15

3faec78

Signed-off-by: Sergey Chvalyuk <[email protected]>

edgao reviewed Aug 30, 2022

View reviewed changes

airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py Show resolved Hide resolved

airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py Show resolved Hide resolved

edgao approved these changes Aug 30, 2022

View reviewed changes

grubberr added 2 commits August 30, 2022 18:42

Merge branch 'master' into grubberr/oncall-412-source-salesforce-2

73092d7

logger.warning -> raise Exception

b1dd625

Signed-off-by: Sergey Chvalyuk <[email protected]>

bazarnov approved these changes Aug 30, 2022

View reviewed changes

grubberr added 2 commits August 30, 2022 21:46

Merge branch 'master' into grubberr/oncall-412-source-salesforce-2

a4e3ca2

fix error message

20c887b

Signed-off-by: Sergey Chvalyuk <[email protected]>

auto-bump connector version [ci skip]

bed814a

grubberr merged commit aab0377 into master Aug 30, 2022

grubberr deleted the grubberr/oncall-412-source-salesforce-2 branch August 30, 2022 22:06

octavia-squidington-iii mentioned this pull request Sep 2, 2022

Bump Airbyte version from 0.40.3 to 0.40.4 #16275

Merged

robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022

🐛 Source SalesForce: better detect API type (airbytehq#16086)

733c07c

Signed-off-by: Sergey Chvalyuk <[email protected]>

vincentkoc mentioned this pull request Dec 20, 2022

Salesforce Stream "Account" cannot be processed by REST or BULK API #20703

Closed

midavadim mentioned this pull request Mar 28, 2023

Source Salesforce: URI too long #14024

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source SalesForce: better detect API type #16086

🐛 Source SalesForce: better detect API type #16086

grubberr commented Aug 29, 2022 •

edited

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

edgao left a comment

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

🐛 Source SalesForce: better detect API type #16086

🐛 Source SalesForce: better detect API type #16086

Conversation

grubberr commented Aug 29, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented Aug 29, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 29, 2022 • edited by github-actions bot Loading

grubberr commented Aug 29, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Aug 29, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

Build Passed

edgao left a comment

Choose a reason for hiding this comment

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

grubberr commented Aug 30, 2022

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

grubberr commented Aug 30, 2022 • edited by github-actions bot Loading

grubberr commented Aug 29, 2022 •

edited

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 29, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading

grubberr commented Aug 30, 2022 •

edited by github-actions bot

Loading