Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📝 Add warning for working with local files and Docker Desktop (/tmp and /private) #21004

Merged
merged 3 commits into from
Jan 5, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/integrations/destinations/local-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ This destination is meant to be used on a local workstation and won't work on Ku

This destination writes data to a directory on the _local_ filesystem on the host running Airbyte. By default, data is written to `/tmp/airbyte_local`. To change this location, modify the `LOCAL_ROOT` environment variable for Airbyte.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

### Sync Overview

#### Output schema
Expand Down
8 changes: 8 additions & 0 deletions docs/integrations/destinations/local-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,13 @@ By default, the `LOCAL_ROOT` env variable in the `.env` file is set `/tmp/airbyt

The local mount is mounted by Docker onto `LOCAL_ROOT`. This means the `/local` is substituted by `/tmp/airbyte_local` by default.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::


### Example:

* If `destination_path` is set to `/local/cars/models`
Expand Down Expand Up @@ -69,4 +76,5 @@ Note: If you are running Airbyte on Windows with Docker backed by WSL2, you have

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.2.12 | 2023-01-04 | [21004](https://github.com/airbytehq/airbyte/pull/21004) | Source File: Adding Docker Desktop file share warning |
| 0.2.11 | 2022-02-14 | [14641](https://github.com/airbytehq/airbyte/pull/14641) | Include lifecycle management |
9 changes: 8 additions & 1 deletion docs/integrations/destinations/sqlite.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ This destination is meant to be used on a local workstation and won't work on Ku

This destination writes data to a file on the _local_ filesystem on the host running Airbyte. By default, data is written to `/tmp/airbyte_local`. To change this location, modify the `LOCAL_ROOT` environment variable for Airbyte.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

### Sync Overview

#### Output schema
Expand Down Expand Up @@ -68,4 +74,5 @@ Note: If you are running Airbyte on Windows with Docker backed by WSL2, you have

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.0 | 2022-07-25 | [15018](https://github.com/airbytehq/airbyte/pull/15018) | New SQLite destination |
| 0.1.1 | 2023-01-04 | [21004](https://github.com/airbytehq/airbyte/pull/21004) | Source File: Adding Docker Desktop file share warning |
| 0.1.0 | 2022-07-25 | [15018](https://github.com/airbytehq/airbyte/pull/15018) | New SQLite destination |
2 changes: 2 additions & 0 deletions docs/integrations/sources/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,12 +133,14 @@ Please see (or add) more at `airbyte-integrations/connectors/source-file/integra
In order to read large files from a remote location, this connector uses the [smart_open](https://pypi.org/project/smart-open/) library. However, it is possible to switch to either [GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) implementations as it is natively supported by the `pandas` library. This choice is made possible through the optional `reader_impl` parameter.

- Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md) so the `URL` should also starts with `/local/`.
- Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.
- The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting.

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :------------------------------------------------------- |:---------------------------------------------------------|
| 0.2.33 | 2023-01-04 | [21004](https://github.com/airbytehq/airbyte/pull/21004) | Source File: Adding Docker Desktop file share warning |
sspaeti marked this conversation as resolved.
Show resolved Hide resolved
| 0.2.32 | 2022-12-21 | [20740](https://github.com/airbytehq/airbyte/pull/20740) | Source File: increase SSH timeout to 60s |
| 0.2.31 | 2022-11-17 | [19567](https://github.com/airbytehq/airbyte/pull/19567) | Source File: bump 0.2.31 |
| 0.2.30 | 2022-11-10 | [19222](https://github.com/airbytehq/airbyte/pull/19222) | Use AirbyteConnectionStatus for "check" command |
Expand Down
6 changes: 6 additions & 0 deletions docs/operator-guides/browsing-output-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ cat ./catalog.json

If you setup a pipeline using one of the local File based destinations \(CSV or JSON\), Airbyte is writing the resulting files containing the data in the special `/local/` directory in the container. By default, this volume is mounted from `/tmp/airbyte_local` on the host machine. So you need to navigate to this [local folder](file:///tmp/airbyte_local/) on the filesystem of the machine running the Airbyte deployment to retrieve the local data files.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

Or, you can also run through docker commands as proxy:

```bash
Expand Down
6 changes: 6 additions & 0 deletions docs/quickstart/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ The destination we are creating is a simple JSON line file, meaning that it will

The resulting files will be located in `/tmp/airbyte_local/json_data`

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

To set it up, just follow the instructions on the screenshot below.

:::info
Expand Down