-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
🎉 New Destination: Implement
Destination Google Sheets using CDK (#…
- Loading branch information
Showing
35 changed files
with
1,804 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
airbyte-integrations/connectors/destination-google-sheets/.dockerignore
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| * | ||
| !Dockerfile | ||
| !main.py | ||
| !destination_google_sheets | ||
| !setup.py |
17 changes: 17 additions & 0 deletions
17
airbyte-integrations/connectors/destination-google-sheets/Dockerfile
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| FROM python:3.9-slim | ||
|
|
||
| # Bash is installed for more convenient debugging. | ||
| RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" | ||
|
|
||
| WORKDIR /airbyte/integration_code | ||
| COPY destination_google_sheets ./destination_google_sheets | ||
| COPY setup.py ./ | ||
| COPY main.py ./ | ||
| RUN pip install . | ||
|
|
||
| ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] | ||
|
|
||
| LABEL io.airbyte.version=0.1.0 | ||
| LABEL io.airbyte.name=airbyte/destination-google-sheets |
123 changes: 123 additions & 0 deletions
123
airbyte-integrations/connectors/destination-google-sheets/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Google Sheets Destination | ||
|
|
||
| This is the repository for the Google Sheets destination connector, written in Python. | ||
| For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/destinations/google-sheets). | ||
|
|
||
| ## Local development | ||
|
|
||
| ### Prerequisites | ||
| **To iterate on this connector, make sure to complete this prerequisites section.** | ||
|
|
||
| #### Minimum Python version required `= 3.9.0` | ||
|
|
||
| #### Build & Activate Virtual Environment and install dependencies | ||
| From this connector directory, create a virtual environment: | ||
| ``` | ||
| python -m venv .venv | ||
| ``` | ||
|
|
||
| This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your | ||
| development environment of choice. To activate it from the terminal, run: | ||
| ``` | ||
| source .venv/bin/activate | ||
| pip install -r requirements.txt | ||
| ``` | ||
| If you are in an IDE, follow your IDE's instructions to activate the virtualenv. | ||
|
|
||
| Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is | ||
| used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`. | ||
| If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything | ||
| should work as you expect. | ||
|
|
||
| #### Building via Gradle | ||
| From the Airbyte repository root, run: | ||
| ``` | ||
| ./gradlew :airbyte-integrations:connectors:destination-google-sheets:build | ||
| ``` | ||
|
|
||
| #### Create credentials | ||
| **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/destinations/google-sheets) | ||
| to generate the necessary credentials. Then create a file `secrets/config_oauth.json` conforming to the `destination_google_sheets/spec.json` file. | ||
| Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information. | ||
| See `integration_tests/sample_config_oauth.json` for a sample config file. | ||
|
|
||
| **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `destination google-sheets test creds` | ||
| and place them into `secrets/config.json`. | ||
|
|
||
| ### Locally running the connector | ||
| ``` | ||
| python main.py spec | ||
| python main.py check --config secrets/config_oauth.json | ||
| python main.py discover --config secrets/config_oauth.json | ||
| cat integration_tests/test_data/messages.txt | python main.py write --config secrets/config_oauth.json --catalog integration_tests/configured_catalog.json | ||
| ``` | ||
|
|
||
| ### Locally running the connector docker image | ||
|
|
||
| #### Build | ||
| First, make sure you build the latest Docker image: | ||
| ``` | ||
| docker build . -t airbyte/destination-google-sheets:dev | ||
| ``` | ||
|
|
||
| You can also build the connector image via Gradle: | ||
| ``` | ||
| ./gradlew :airbyte-integrations:connectors:destination-google-sheets:airbyteDocker | ||
| ``` | ||
| When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in | ||
| the Dockerfile. | ||
|
|
||
| #### Run | ||
| Then run any of the connector commands as follows: | ||
| ``` | ||
| docker run --rm airbyte/destination-google-sheets:dev spec | ||
| docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-google-sheets:dev check --config /secrets/config_oauth.json | ||
| # messages.txt is a file containing line-separated JSON representing AirbyteMessages | ||
| cat integration_tests/test_data/messages.txt | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-google-sheets:dev write --config /secrets/config_oauth.json --catalog /integration_tests/configured_catalog.json | ||
| ``` | ||
| ## Testing | ||
| Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named. | ||
| First install test dependencies into your virtual environment: | ||
| ``` | ||
| pip install .[tests] | ||
| ``` | ||
| ### Unit Tests | ||
| To run unit tests locally, from the connector directory run: | ||
| ``` | ||
| python -m pytest unit_tests | ||
| ``` | ||
|
|
||
| ### Integration Tests | ||
| There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all destination connectors) and custom integration tests (which are specific to this connector). | ||
| #### Custom Integration tests | ||
| Place custom tests inside `integration_tests/` folder, then, from the connector root, run | ||
| ``` | ||
| python -m pytest integration_tests | ||
| ``` | ||
| #### Acceptance Tests | ||
| Coming soon: | ||
|
|
||
| ### Using gradle to run tests | ||
| All commands should be run from airbyte project root. | ||
| To run unit tests: | ||
| ``` | ||
| ./gradlew :airbyte-integrations:connectors:destination-google-sheets:unitTest | ||
| ``` | ||
| To run acceptance and custom integration tests: | ||
| ``` | ||
| ./gradlew :airbyte-integrations:connectors:destination-google-sheets:integrationTest | ||
| ``` | ||
|
|
||
| ## Dependency Management | ||
| All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development. | ||
| We split dependencies between two groups, dependencies that are: | ||
| * required for your connector to work need to go to `MAIN_REQUIREMENTS` list. | ||
| * required for the testing need to go to `TEST_REQUIREMENTS` list | ||
|
|
||
| ### Publishing a new version of the connector | ||
| You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? | ||
| 1. Make sure your changes are passing unit and integration tests. | ||
| 1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)). | ||
| 1. Create a Pull Request. | ||
| 1. Pat yourself on the back for being an awesome contributor. | ||
| 1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. |
8 changes: 8 additions & 0 deletions
8
airbyte-integrations/connectors/destination-google-sheets/build.gradle
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| plugins { | ||
| id 'airbyte-python' | ||
| id 'airbyte-docker' | ||
| } | ||
|
|
||
| airbytePython { | ||
| moduleDirectory 'destination_google_sheets' | ||
| } |
8 changes: 8 additions & 0 deletions
8
...e-integrations/connectors/destination-google-sheets/destination_google_sheets/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # | ||
| # Copyright (c) 2021 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
|
|
||
| from .destination import DestinationGoogleSheets | ||
|
|
||
| __all__ = ["DestinationGoogleSheets"] |
113 changes: 113 additions & 0 deletions
113
...yte-integrations/connectors/destination-google-sheets/destination_google_sheets/buffer.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # | ||
| # Copyright (c) 2021 Airbyte, Inc., all rights reserved. | ||
| # | ||
|
|
||
|
|
||
| from typing import Any, Mapping | ||
|
|
||
| from airbyte_cdk import AirbyteLogger | ||
| from airbyte_cdk.models import AirbyteStream | ||
|
|
||
|
|
||
| class WriteBufferMixin: | ||
|
|
||
| # Default instance of AirbyteLogger | ||
| logger = AirbyteLogger() | ||
| # interval after which the records_buffer should be cleaned up for selected stream | ||
| flush_interval = 1000 | ||
|
|
||
| def __init__(self): | ||
| # Buffer for input records | ||
| self.records_buffer = {} | ||
| # Placeholder for streams metadata | ||
| self.stream_info = {} | ||
|
|
||
| @property | ||
| def default_missing(self) -> str: | ||
| """ | ||
| Default value for missing keys in record stream, compared to configured_stream catalog. | ||
| Overwrite if needed. | ||
| """ | ||
| return "" | ||
|
|
||
| def init_buffer_stream(self, configured_stream: AirbyteStream): | ||
| """ | ||
| Saves important stream's information for later use. | ||
| Particulary, creates the data structure for `records_stream`. | ||
| Populates `stream_info` placeholder with stream metadata information. | ||
| """ | ||
| stream = configured_stream.stream | ||
| self.records_buffer[stream.name] = [] | ||
| self.stream_info[stream.name] = { | ||
| "headers": sorted(list(stream.json_schema.get("properties").keys())), | ||
| "is_set": False, | ||
| } | ||
|
|
||
| def add_to_buffer(self, stream_name: str, record: Mapping): | ||
| """ | ||
| Populates input records to `records_buffer`. | ||
| 1) normalizes input record | ||
| 2) coerces normalized record to str | ||
| 3) gets values as list of record values from record mapping. | ||
| """ | ||
|
|
||
| norm_record = self._normalize_record(stream_name, record) | ||
| norm_values = list(map(str, norm_record.values())) | ||
| self.records_buffer[stream_name].append(norm_values) | ||
|
|
||
| def clear_buffer(self, stream_name: str): | ||
| """ | ||
| Cleans up the `records_buffer` values, belonging to input stream. | ||
| """ | ||
| self.records_buffer[stream_name].clear() | ||
|
|
||
| def _normalize_record(self, stream_name: str, record: Mapping) -> Mapping[str, Any]: | ||
| """ | ||
| Updates the record keys up to the input configured_stream catalog keys. | ||
| Handles two scenarios: | ||
| 1) when record has less keys than catalog declares (undersetting) | ||
| 2) when record has more keys than catalog declares (oversetting) | ||
| Returns: alphabetically sorted, catalog-normalized Mapping[str, Any]. | ||
| EXAMPLE: | ||
| - UnderSetting: | ||
| * Catalog: | ||
| - has 3 entities: | ||
| [ 'id', 'key1', 'key2' ] | ||
| ^ | ||
| * Input record: | ||
| - missing 1 entity, compare to catalog | ||
| { 'id': 123, 'key2': 'value' } | ||
| ^ | ||
| * Result: | ||
| - 'key1' has been added to the record, because it was declared in catalog, to keep the data structure. | ||
| {'id': 123, 'key1': '', {'key2': 'value'} } | ||
| ^ | ||
| - OverSetting: | ||
| * Catalog: | ||
| - has 3 entities: | ||
| [ 'id', 'key1', 'key2', ] | ||
| ^ | ||
| * Input record: | ||
| - doesn't have entity 'key1' | ||
| - has 1 more enitity, compare to catalog 'key3' | ||
| { 'id': 123, ,'key2': 'value', 'key3': 'value' } | ||
| ^ ^ | ||
| * Result: | ||
| - 'key1' was added, because it expected be the part of the record, to keep the data structure | ||
| - 'key3' was dropped, because it was not declared in catalog, to keep the data structure | ||
| { 'id': 123, 'key1': '', 'key2': 'value', } | ||
| ^ ^ | ||
| """ | ||
| headers = self.stream_info[stream_name]["headers"] | ||
| # undersetting scenario | ||
| [record.update({key: self.default_missing}) for key in headers if key not in record.keys()] | ||
| # oversetting scenario | ||
| [record.pop(key) for key in record.copy().keys() if key not in headers] | ||
|
|
||
| return dict(sorted(record.items(), key=lambda x: x[0])) |
Oops, something went wrong.