Skip to content

Commit

Permalink
📖 improve salesforce docs & reorder properties in the spec (#10679)
Browse files Browse the repository at this point in the history
  • Loading branch information
sherifnada authored Feb 28, 2022
1 parent 2bba529 commit 5691434
Show file tree
Hide file tree
Showing 5 changed files with 73 additions and 44 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -669,7 +669,7 @@
- name: Salesforce
sourceDefinitionId: b117307c-14b6-41aa-9422-947e34922962
dockerRepository: airbyte/source-salesforce
dockerImageTag: 1.0.0
dockerImageTag: 1.0.1
documentationUrl: https://docs.airbyte.io/integrations/sources/salesforce
icon: salesforce.svg
sourceType: api
Expand Down
32 changes: 20 additions & 12 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7064,7 +7064,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-salesforce:1.0.0"
- dockerImage: "airbyte/source-salesforce:1.0.1"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/salesforce"
connectionSpecification:
Expand All @@ -7077,25 +7077,37 @@
- "refresh_token"
additionalProperties: true
properties:
is_sandbox:
title: "Sandbox"
description: "Whether the app is in a Salesforce sandbox or not. If you\
\ do not know what this is, assume it as false. We provide more info on\
\ this field in the <a href=\"https://docs.airbyte.com/integrations/sources/salesforce#is_sandbox\"\
>docs</a>."
type: "boolean"
default: false
order: 1
auth_type:
type: "string"
const: "Client"
client_id:
title: "Client ID"
description: "The Client ID of your Salesforce developer application."
type: "string"
order: 2
client_secret:
title: "Client Secret"
description: "The Client Secret of your Salesforce developer application."
type: "string"
airbyte_secret: true
order: 3
refresh_token:
title: "Refresh Token"
description: "Salesforce Refresh Token used for Airbyte to access your Salesforce\
\ account. If you don't know what this is, follow this <a href=\"https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b\"\
>guide</a> to retrieve it."
type: "string"
airbyte_secret: true
order: 4
start_date:
title: "Start Date"
description: "Date in the format 2017-01-25. Any data before this date will\
Expand All @@ -7107,16 +7119,10 @@
examples:
- "2021-07-25"
- "2021-07-25T00:00:00Z"
is_sandbox:
title: "Sandbox"
description: "Whether the app is in a Salesforce sandbox or not. If you\
\ do not know what this is, assume it as false. We provide more info on\
\ this field in the <a href=\"https://docs.airbyte.com/integrations/sources/salesforce#is_sandbox\"\
>docs</a>."
type: "boolean"
default: false
order: 5
streams_criteria:
type: "array"
order: 6
items:
type: "object"
required:
Expand All @@ -7135,13 +7141,15 @@
- "ends not with"
- "not contains"
- "not exacts"
order: 1
default: "contains"
value:
type: "string"
title: "Search value"
title: "Streams filter criteria"
description: "Add selection criteria for streams to get only streams that\
\ are relevant to you"
order: 2
title: "Salesforce Object filtering criteria (Optional)"
description: "Add selection criteria for streams to read only the Salesforce\
\ objects which are relevant to you."
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ COPY source_salesforce ./source_salesforce
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=1.0.0
LABEL io.airbyte.version=1.0.1
LABEL io.airbyte.name=airbyte/source-salesforce
Original file line number Diff line number Diff line change
Expand Up @@ -7,42 +7,48 @@
"required": ["client_id", "client_secret", "refresh_token"],
"additionalProperties": true,
"properties": {
"is_sandbox": {
"title": "Sandbox",
"description": "Whether the app is in a Salesforce sandbox or not. If you do not know what this is, assume it as false. We provide more info on this field in the <a href=\"https://docs.airbyte.com/integrations/sources/salesforce#is_sandbox\">docs</a>.",
"type": "boolean",
"default": false,
"order": 1
},
"auth_type": {
"type": "string",
"const": "Client"
},
"client_id": {
"title": "Client ID",
"description": "The Client ID of your Salesforce developer application.",
"type": "string"
"type": "string",
"order": 2
},
"client_secret": {
"title": "Client Secret",
"description": "The Client Secret of your Salesforce developer application.",
"type": "string",
"airbyte_secret": true
"airbyte_secret": true,
"order": 3
},
"refresh_token": {
"title": "Refresh Token",
"description": "Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow this <a href=\"https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b\">guide</a> to retrieve it.",
"type": "string",
"airbyte_secret": true
"airbyte_secret": true,
"order": 4
},
"start_date": {
"title": "Start Date",
"description": "Date in the format 2017-01-25. Any data before this date will not be replicated. This field uses the \"updated\" field if available, otherwise the \"created\" fields if they are available for a stream. If not set, then by default all your data is replicated.",
"type": "string",
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z|[0-9]{4}-[0-9]{2}-[0-9]{2}$",
"examples": ["2021-07-25", "2021-07-25T00:00:00Z"]
},
"is_sandbox": {
"title": "Sandbox",
"description": "Whether the app is in a Salesforce sandbox or not. If you do not know what this is, assume it as false. We provide more info on this field in the <a href=\"https://docs.airbyte.com/integrations/sources/salesforce#is_sandbox\">docs</a>.",
"type": "boolean",
"default": false
"examples": ["2021-07-25", "2021-07-25T00:00:00Z"],
"order": 5
},
"streams_criteria": {
"type": "array",
"order": 6,
"items": {
"type": "object",
"required": ["criteria", "value"],
Expand All @@ -60,16 +66,18 @@
"not contains",
"not exacts"
],
"order": 1,
"default": "contains"
},
"value": {
"type": "string",
"title": "Search value"
"title": "Search value",
"order": 2
}
}
},
"title": "Streams filter criteria",
"description": "Add selection criteria for streams to get only streams that are relevant to you"
"title": "Salesforce Object filtering criteria (Optional)",
"description": "Add selection criteria for streams to read only the Salesforce objects which are relevant to you."
}
}
},
Expand Down
47 changes: 30 additions & 17 deletions docs/integrations/sources/salesforce.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,7 @@

The Salesforce source supports both `Full Refresh` and `Incremental` syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run.

The Connector supports `Custom Fields` for each of their available streams

### Output schema

Several output streams are available from this source. A list of these streams can be found below in the [Streams](salesforce.md#streams) section.
The Connector supports replicating both standard and custom Salesforce objects.

### Features

Expand All @@ -20,20 +16,37 @@ Several output streams are available from this source. A list of these streams c
| Namespaces | No |

#### Incremental Deletes Sync
We can retrieve deleted records in Salesforce. It is available only for streams with the `IsDeleted` field in the schema.
If the record was deleted, Salesforce set `IsDeleted=True` and in the next incremental sync, we read this record.
In case, when the record was created, synced, then updated and deleted, synced one more time, we retrieve the record with updated data and `IsDeleted=True`.
This connector retrieves deleted records from Salesforce. For the streams which support it, a deleted record will be marked with the field `isDeleted=true` value.

### Performance considerations

The connector is restricted by daily Salesforce rate limiting.
The connector uses as much rate limit as it can every day, then ends the sync early with success status and continues the sync from where it left the next time.
Note that, picking up from where it ends will work only for incremental sync.

## Getting started
## Getting Started (Airbyte Cloud)
#### Sandbox accounts

If you log in using at [https://login.salesforce.com](https://login.salesforce.com), then your account is not a sandbox. If you log in at [https://test.salesforce.com](https://test.salesforce.com) then it's a sandbox.

If this is Greek to you, then you are likely not using a sandbox account.

### Requirements

* Salesforce Account
* Dedicated Salesforce user (optional)

**Note**: We recommend creating a new Salesforce user, restricted, read-only OAuth credentials specifically for Airbyte access. In addition, you can restrict access to only the data and streams you need by creating a profile in Salesforce and assigning it to the user.

### Setup guide

1. Toggle whether your Salesforce account is a Sandbox account or a live account.
2. Click `Authenticate your Salesforce account` to sign in with Salesforce and authorize your account.
3. Fill in the rest of the details.
4. You should be ready to sync data.

## Getting started (Airbyte OSS)
### Requirements
* Salesforce Account
* Salesforce OAuth credentials
* Dedicated Salesforce user (optional)
Expand All @@ -42,25 +55,24 @@ Note that, picking up from where it ends will work only for incremental sync.

### Setup guide

#### Sandbox accounts
If you log in using at [https://login.salesforce.com](https://login.salesforce.com), then your account is not a sandbox. If you log in at [https://test.salesforce.com](https://test.salesforce.com) then it's a sandbox.

If this is Greek to you, then you are likely not using a sandbox account.
We recommend the following [walkthrough](https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b) **while keeping in mind the edits we suggest below** for setting up a Salesforce app that can pull data from Salesforce and locating the credentials you need to provide to Airbyte.

Suggested edits:

1. If your salesforce URL does not take the form `X.salesforce.com`, use your actual Salesforce domain name. For example, if your Salesforce URL is `awesomecompany.force.com` then use that instead of `awesomecompany.salesforce.com`.
2. When running a `curl` command, always run it with the `-L` option to follow any redirects.

#### is\_sandbox

If you log in using at [https://login.salesforce.com](https://login.salesforce.com), then the value is false. If you log in at [https://test.salesforce.com](https://test.salesforce.com) then the value should be true. If this is Greek to you, then this value should probably be false.

## Streams

**Note**: The connector supports reading not only standard streams, but also reading `Custom Objects`.
**Note**: The connector supports reading both standard streams and Custom Objects from Salesforce.

We fetch and handle all the possible & available streams dynamically based on:
- User Role & Permissions to read & fetch objects and their data (administration question);
- Whether or not, the stream has the queryable property is set to true, which means it's available to fetch the data, otherwise we will hit the 400 - bad request, or 503 - forbidden in response;
- No other restrictions applied, so we should have the full list of available streams as much as possible (both standard objects & custom ones along with standard and custom fields).
- User Role & Permissions to read & fetch objects and their data
- Whether or not the stream has the queryable property set to true. Queryable streams are available to be fetched via the API. If you cannot see your object available via Airbyte, please ensure it is API-accessible to the user you used for authenticating into Airbyte

**Note**: Using the BULK API is not possible to receive data from the following streams:

Expand Down Expand Up @@ -88,6 +100,7 @@ We fetch and handle all the possible & available streams dynamically based on:

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- |:---------------------------------------------------------------------------------------------------------------------------------|
| 1.0.1 | 2022-02-27 | [10679](https://github.com/airbytehq/airbyte/pull/10679) | Reorganize input parameter order on the UI |
| 1.0.0 | 2022-02-27 | [10516](https://github.com/airbytehq/airbyte/pull/10516) | Speed up schema discovery by using parallelism |
| 0.1.23 | 2022-02-10 | [10141](https://github.com/airbytehq/airbyte/pull/10141) | Processing of failed jobs |
| 0.1.22 | 2022-02-02 | [10012](https://github.com/airbytehq/airbyte/pull/10012) | Increase CSV field_size_limit |
Expand Down

0 comments on commit 5691434

Please sign in to comment.