-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update getting-started-with-airbyte-cloud.md #13741
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,7 +16,7 @@ To use Airbyte Cloud: | |
A workspace lets you collaborate with team members and share resources across your team under a shared billing account. | ||
::: | ||
|
||
You will be greeted with an onboarding tutorial to help you set up your first connection. If you haven’t set up a connection on Airbyte Cloud before, we highly recommend following the tutorial. If you are familiar with the connection setup process, click **Skip onboarding** and follow this guide to set up your next connection. | ||
You will be greeted with an onboarding tutorial to help you set up your first connection. If you haven’t set up a connection on Airbyte Cloud before, we highly recommend following the tutorial. If you are familiar with the connection setup process, click **Skip Onboarding** and follow this guide to set up your next connection. | ||
|
||
## Set up a source | ||
|
||
|
@@ -26,13 +26,10 @@ A source is an API, file, database, or data warehouse that you want to ingest da | |
|
||
To set up a source: | ||
|
||
1. On the Airbyte Cloud dashboard, click **Sources** in the left navigation bar. | ||
2. In the top right corner, click **+ new source**. | ||
3. On the Set up the source page, enter a name for your source. | ||
4. From the Source type dropdown, select the source you want to set up. | ||
5. The fields relevant to your source are displayed. | ||
Click **Setup Guide** for help with filling in the fields for your selected source. | ||
6. Click **Set up source**. | ||
1. On the Airbyte Cloud dashboard, click **Sources** and then click **+ New source**. | ||
2. On the Set up the source page, select the source you want to set up from the **Source type** dropdown. | ||
3. The fields relevant to your source are displayed. The Setup Guide provides information to help you fill out the fields for your selected source. | ||
4. Click **Set up source**. | ||
|
||
## Set up a destination | ||
|
||
|
@@ -42,13 +39,10 @@ A destination is a data warehouse, data lake, database, or an analytics tool whe | |
|
||
To set up a destination: | ||
|
||
1. On the Airbyte Cloud dashboard, click **Destinations** in the left navigation bar. | ||
2. In the top right corner, click **+ New destination**. | ||
3. On the Set up the destination page, enter a name for your destination. | ||
4. From the Destination type dropdown, select the destination you want to set up. | ||
5. The fields relevant to your destination are displayed. | ||
Click **Setup Guide** for help with filling in the fields for your selected destination. | ||
6. Click **Set up destination**. | ||
1. On the Airbyte Cloud dashboard, click **Destinations** and then click **+ New destination**. | ||
2. On the Set up the destination page, select the destination you want to set up from the **Destination type** dropdown. | ||
3. The fields relevant to your destination are displayed. The Setup Guide provides information to help you fill out the fields for your selected destination. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as above |
||
4. Click **Set up destination**. | ||
|
||
## Set up a connection | ||
|
||
|
@@ -66,9 +60,9 @@ Setting up a connection involves configuring the following parameters: | |
</td> | ||
</tr> | ||
<tr> | ||
<td>Sync schedule | ||
<td>Replication frequency | ||
</td> | ||
<td>When should a data sync be triggered? | ||
<td>How often should the data sync? | ||
</td> | ||
</tr> | ||
<tr> | ||
|
@@ -102,23 +96,24 @@ For more information, see [Connections and Sync Modes](../understanding-airbyte/ | |
|
||
To set up a connection: | ||
|
||
1. On the Airbyte Cloud dashboard, click **Connections** in the left navigation bar. | ||
2. In the top right corner, click **+ new connection**. | ||
3. On the New Connection page, select a source: | ||
- To use an existing source, select your desired source from the Source dropdown. Click **Use existing source**. | ||
1. On the Airbyte Cloud dashboard, click **Connections** and then click **+ New connection**. | ||
2. On the New connection page, select a source: | ||
- To use an existing source, select your desired source from the **Source** dropdown. Click **Use existing source**. | ||
|
||
- To set up a new source, enter a name for the new source and select the source from the Source type dropdown. The fields relevant to your source are displayed. Click **Setup Guide** for help with filling in the fields for your selected source. Click **Set up source**. | ||
- To set up a new source, select the source you want to set up from the **Source type** dropdown. The fields relevant to your source are displayed. The Setup Guide provides information to help you fill out the fields for your selected source. Click **Set up source**. | ||
|
||
4. Select a destination: | ||
- To use an existing destination, select your desired destination from the Destination dropdown. Click **Use existing destination**. | ||
- To set up a new destination, enter a name for the new destination and select the destination from the Destination type dropdown. The fields relevant to your destination are displayed. Click **Setup Guide** for help with filling in the fields for your selected source. Click **Set up destination**. | ||
3. Select a destination: | ||
- To use an existing destination, select your desired destination from the **Destination** dropdown. Click **Use existing destination**. | ||
- To set up a new destination, select the source you want to set up from the **Destination type** dropdown. The fields relevant to your destination are displayed. The Setup Guide provides information to help you fill out the fields for your selected source. Click **Set up destination**. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: replace source with destination in "select the source you want to set up" and "selected source" |
||
|
||
The Set up the connection page is displayed. | ||
5. From the **Replication frequency** dropdown, select how often you want the data to sync from the source to the destination. | ||
4. From the **Replication frequency** dropdown, select how often you want the data to sync from the source to the destination. | ||
|
||
**Note:** The default replication frequency is 24 hours. | ||
**Note:** The default replication frequency is **Every 24 hours**. | ||
|
||
6. From the **Destination Namespace** dropdown, select the format in which you want the data to stored in the destination: | ||
5. From the **Destination Namespace** dropdown, select the format in which you want the data to stored in the destination: | ||
|
||
**Note:** The default namespace is **Mirror source structure**. | ||
|
||
<table> | ||
<tr> | ||
|
@@ -130,34 +125,34 @@ To set up a connection: | |
<tr> | ||
<td>Mirror source structure | ||
</td> | ||
<td>Some sources (for example, databases) provide namespace information for a stream. If a source provides the namespace information, the destination will reproduce the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option | ||
<td>Some sources (for example, databases) provide namespace information for a stream. If a source provides the namespace information, the destination will reproduce the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option. | ||
</td> | ||
</tr> | ||
<tr> | ||
<td>Destination default | ||
</td> | ||
<td>All streams will be replicated and stored in the default namespace defined on the destination settings page. For settings for popular destinations, see<a href="https://docs.airbyte.com/understanding-airbyte/namespaces#destination-connector-settings"> Destination Connector Settings</a> | ||
<td>All streams will be replicated and stored in the default namespace defined on the Destination Settings page. For more information, see<a href="https://docs.airbyte.com/understanding-airbyte/namespaces#destination-connector-settings"> Destination Connector Settings</a>. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: we usually don't put periods for sentences that end with hyperlinks |
||
</td> | ||
</tr> | ||
<tr> | ||
<td>Custom format | ||
</td> | ||
<td>All streams will be replicated and stored in a custom format. See<a href="https://docs.airbyte.com/understanding-airbyte/namespaces#custom-format"> Custom format</a> for more details | ||
<td>All streams will be replicated and stored in a custom format. See<a href="https://docs.airbyte.com/understanding-airbyte/namespaces#custom-format"> Custom format</a> for more details. | ||
</td> | ||
</tr> | ||
</table> | ||
|
||
|
||
:::tip | ||
To better understand the destination namespace configurations, see [Destination Namespace example](../understanding-airbyte/namespaces.md#examples) | ||
To better understand the destination namespace configurations, see [Destination Namespace example](../understanding-airbyte/namespaces.md#examples). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same as above |
||
::: | ||
|
||
7. (Optional) In the **Destination Stream Prefix (Optional)** field, add a prefix to stream names (for example, adding a prefix `airbyte_` renames `projects` to `airbyte_projects`). | ||
8. (Optional) Click **Refresh schema** if you had previously triggered a sync with a subset of tables in the stream and now want to see all the tables in the stream. | ||
9. Activate the streams you want to sync: | ||
- (Optional) If your source has multiple tables, type the name of the stream you want to enable in the **Search box**. | ||
6. (Optional) In the **Destination Stream Prefix (Optional)** field, add a prefix to stream names (for example, adding a prefix `airbyte_` renames `projects` to `airbyte_projects`). | ||
7. (Optional) Click **Refresh schema** if you had previously triggered a sync with a subset of tables in the stream and now want to see all the tables in the stream. | ||
8. Activate the streams you want to sync: | ||
- (Optional) If your source has multiple tables, type the name of the stream you want to enable in the **Search stream name** search box. | ||
- (Optional) To configure the sync settings for multiple streams, select the checkbox next to the desired streams, configure the settings in the purple box, and click **Apply**. | ||
10. Configure the sync settings and click **Set up connection**: | ||
9. Configure the sync settings: | ||
1. Toggle the **Sync** button to enable sync for the stream. | ||
2. **Source:** | ||
1. **Namespace**: The database schema of your source tables (auto-populated for your source) | ||
|
@@ -167,30 +162,32 @@ To better understand the destination namespace configurations, see [Destination | |
For the source: | ||
|
||
* Select **Full Refresh** to copy the entire dataset each time you sync | ||
* Select **Incremental only** to replicate only the new or modified data | ||
* Select **Incremental** to replicate only the new or modified data | ||
|
||
For the destination: | ||
|
||
* Select **Overwrite** to erase the old data and replace it completely | ||
* Select **Append** to capture changes to your table | ||
|
||
**Note:** This creates duplicate records | ||
* Select **Deduped + History** to mirror your source while keeping records unique | ||
**Note:** Some sync modes may not yet be available for your source or destination | ||
* Select **Deduped + history** to mirror your source while keeping records unique | ||
|
||
**Note:** Some sync modes may not yet be available for your source or destination | ||
|
||
4. **Cursor field**: Used in incremental sync mode to determine which records to sync. Airbyte pre-selects the Cursor field for you (example: updated date). If you have multiple cursor fields, select the one you want. | ||
5. **Primary key**: Used in Deduped and History modes to determine the unique identifier. | ||
6. **Destination**: | ||
5. **Primary key**: Used in Deduped + history sync mode to determine the unique identifier. | ||
6. **Destination**: | ||
- **Namespace:** The database schema of your destination tables. | ||
- **Stream name:** The final table name in destination. | ||
11. If the sync is successful, a success message is displayed. | ||
10. Click **Set up connection**. | ||
11. Airbyte tests the connection. If the sync is successful, the Connection page will display. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: the Connection page is displayed. |
||
|
||
## Verify the connection | ||
|
||
To verify the sync by checking the logs: | ||
Verify the sync by checking the logs: | ||
|
||
1. On the Airbyte Cloud dashboard, click **Connections** in the left navigation bar. The list of connections is displayed. Click on the connection you just set up. | ||
2. The sync history is displayed. Click on the first log in the sync history. | ||
1. On the Airbyte Cloud dashboard, click **Connections**. The list of connections is displayed. Click on the connection you just set up. | ||
2. The Sync History is displayed. Click on the first log in the sync history. | ||
3. Check the data at your destination. If you added a Destination Stream Prefix while setting up the connection, make sure to search for the stream name with the prefix. | ||
|
||
## Allowlist IP address | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Step 3 isn't really a step since there's no action item for the user. You can merge it with the previous step by adding a newline with the
tag.