Source Salesforce: URI too long

[lgomezm]

Environment

• Airbyte version: 0.39.7-alpha.
• OS Version / Instance: GCP n2-standard-4.
• Deployment: Kubernetes engine on GCP.
• Source Connector and version: Salesforce 1.0.9
• Destination Connector and version: Google Pub/Sub 1.0.4
• Step where error happened: Sync job.

Current Behavior

After some incremental updates, one of our Salesforce -> Google Pub/Sub connections started to fail to get data from Salesforce with the following error:

2022-06-14 19:35:32 �[44msource�[0m > Read 3315 records from Account stream
2022-06-14 19:35:32 �[44msource�[0m > Syncing stream: Contact 
2022-06-14 19:35:32 �[44msource�[0m > Encountered an exception while reading stream SourceSalesforce
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/airbyte/integration_code/source_salesforce/source.py", line 111, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 215, in _read_incremental
    for record_counter, record_data in enumerate(records, start=1):
  File "/airbyte/integration_code/source_salesforce/streams.py", line 142, in read_records
    error_data = error.response.json()[0]
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <h1>Bad Message 414</h1><pre>reason: URI Too Long</pre>: 0
2022-06-14 19:35:32 �[44msource�[0m > [Errno Expecting value] <h1>Bad Message 414</h1><pre>reason: URI Too Long</pre>: 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Attempting to reset data to do full refresh also resulted in the same error.

Expected Behavior

The sync job should successfully sync data from Salesforce.

Logs

logs-1575317.txt

Steps to Reproduce

I am not sure exactly on how to reproduce this error, but I think, since Salesforce builds the URI request parameters out of the parameters, I believe it can end up being a really long URI. I believe the easiest way of reproducing would be having an entity with a lot of fields. Maybe increasing the list of fields with custom fields can help.

1. Have a Salesforce account with an entity having a long list of custom fields.
2. Create a connection from that Salesforce account to any destination.
3. The error occurs.

Are you willing to submit a PR?

Yes.

[lgomezm]

This could be related: I found out that other connections fail with a similar error for headers:
logs-370562.txt

[midavadim]

I have checked logs, and errors are related to "Contact" Stream:

1. Bad Message 431

   reason: Request Header Fields Too Large

2. [Errno Expecting value]

   Bad Message 414

   reason: URI Too Long

   : 0

I was not able to reproduce this error locally.

I have checked the change history and since that time there were a few PRs that tries to solve mentioned errors by adding checking for API limits:
https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta/salesforce_app_limits_cheatsheet/salesforce_app_limits_platform_api.htm

like:
#22896
#16086

So my conclusion is that this problem should be fixed for now. So I close this issue.