Skip to content

Commit

Permalink
[UPD] rename from ftp to sftp bulk
Browse files Browse the repository at this point in the history
Signed-off-by: Henri Blancke <[email protected]>
  • Loading branch information
henriblancke committed Oct 19, 2022
1 parent 02b04ac commit 79580cd
Show file tree
Hide file tree
Showing 34 changed files with 32 additions and 30 deletions.
1 change: 1 addition & 0 deletions airbyte-integrations/builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@
| Salesloft | [![source-salesloft](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-salesloft%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-salesloft) |
| Sendgrid | [![source-sendgrid](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-sendgrid%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-sendgrid) |
| Sentry | [![source-sentry](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-sentry%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-sentry) |
| SFTP Bulk | [![source-sftp-bulk](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-sftp-bulk%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-sftp-bulk) |
| Shopify | [![source-shopify](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-shopify%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-shopify) |
| Slack | [![source-slack](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-slack%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-slack) |
| Smartsheets | [![source-smartsheets](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-smartsheets%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-smartsheets) |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
*
!Dockerfile
!main.py
!source_ftp
!source_sftp_bulk
!setup.py
!secrets
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ FROM python:3.9-slim
# RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*

WORKDIR /airbyte/integration_code
COPY source_ftp ./source_ftp
COPY source_sftp_bulk ./source_sftp_bulk
COPY main.py ./
COPY setup.py ./
RUN pip install .
Expand All @@ -14,4 +14,4 @@ ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-ftp
LABEL io.airbyte.name=airbyte/source-sftp-bulk
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Ftp Source
# SFTP Bulk Source

This is the repository for the Ftp source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/ftp).
This is the repository for the FTP source connector, written in Python, that helps you bulk ingest files with the same data format from an FTP server into a single stream.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/sftp-bulk).

## Local development

Expand Down Expand Up @@ -32,12 +32,12 @@ should work as you expect.
#### Building via Gradle
From the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-ftp:build
./gradlew :airbyte-integrations:connectors:source-sftp-bulk:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/ftp)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_ftp/spec.yaml` file.
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/sftp-bulk)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_sftp_bulk/spec.json` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

Expand All @@ -57,23 +57,23 @@ python main.py read --config secrets/config.json --catalog integration_tests/con
#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-ftp:dev
docker build . -t airbyte/source-sftp-bulk:dev
```

You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:source-ftp:airbyteDocker
./gradlew :airbyte-integrations:connectors:source-sftp-bulk:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-ftp:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-ftp:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-ftp:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-ftp:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
docker run --rm airbyte/source-sftp-bulk:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sftp-bulk:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sftp-bulk:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-sftp-bulk:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
Expand Down Expand Up @@ -107,11 +107,11 @@ To run your integration tests with docker
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-ftp:unitTest
./gradlew :airbyte-integrations:connectors:source-sftp-bulk:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-ftp:integrationTest
./gradlew :airbyte-integrations:connectors:source-sftp-bulk:integrationTest
```

## Dependency Management
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-ftp:dev
connector_image: airbyte/source-sftp-bulk:dev
tests:
spec:
- spec_path: "source_ftp/spec.json"
- spec_path: "source_sftp_bulk/spec.json"
timeout_seconds: 60
connection:
- config_path: "integration_tests/valid_config.json"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ plugins {
}

airbytePython {
moduleDirectory 'source_ftp_singer'
moduleDirectory 'source_sftp_bulk'
}
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

from airbyte_cdk.models import AirbyteStream, ConfiguredAirbyteCatalog, ConfiguredAirbyteStream, DestinationSyncMode, SyncMode, Type, Status

from source_ftp import SourceFtp
from source_sftp_bulk import SourceFtp

pytest_plugins = ("source_acceptance_test.plugin",)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import sys

from airbyte_cdk.entrypoint import launch
from source_ftp import SourceFtp
from source_sftp_bulk import SourceFtp

if __name__ == "__main__":
source = SourceFtp()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
]

setup(
name="source_ftp",
description="Source implementation for Ftp.",
name="source_sftp_bulk",
description="Source implementation for SFTP Bulk.",
author="Airbyte",
author_email="[email protected]",
packages=find_packages(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#

from source_ftp.client import SFTPClient
from source_sftp_bulk.client import SFTPClient


def test_get_files_matching_pattern_match():
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#

from source_ftp import SourceFtp
from source_sftp_bulk import SourceFtp

source = SourceFtp()

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ For more information about the grading system, see [Product Release Stages](http
| [SearchMetrics](./sources/search-metrics.md) | Alpha | No |
| [Sendgrid](sources/sendgrid.md) | Beta | Yes |
| [Sentry](sources/sentry.md) | Generally Available | Yes |
| [SFTP Bulk](sources/sftp-bulk.md) | Alpha | Yes |
| [SFTP](sources/sftp.md) | Alpha | Yes |
| [Shopify](sources/shopify.md) | Alpha | No |
| [Short.io](sources/shortio.md) | Alpha | Yes |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# FTP
# SFTP Bulk
This page contains the setup guide and reference information for the FTP source connector.

This connector allows you to:
Expand All @@ -14,7 +14,7 @@ This connector allows you to:
* Username-Password/Public Key Access Rights

## Setup guide
### Step 1: Set up FTP
### Step 1: Set up SFTP
1. Use your username/password credential to connect the server.
2. Alternatively generate Public Key Access

Expand All @@ -25,10 +25,10 @@ Private key stays with the user (and only there), while the public key is sent t
Server stores the public key (and "marks" it as authorized).
Server will now allow access to anyone who can prove they have the corresponding private key.

### Step 2: Set up the FTP connector in Airbyte
### Step 2: Set up the SFTP connector in Airbyte

1. In the left navigation bar, click **`Sources`**. In the top-right corner, click **+new source**.
2. On the Set up the source page, enter the name for the FTP connector and select **FTP** from the Source type dropdown.
2. On the Set up the source page, enter the name for the FTP connector and select **SFTP Bulk** from the Source type dropdown.
3. Enter your `User Name`, `Host Address`, `Port`
4. Enter authentication details for the FTP server (`Password` and/or `Private Key`)
5. Choose a `File type`
Expand Down

0 comments on commit 79580cd

Please sign in to comment.