Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] SSH tunnel support for OLTP DB sources & destinations #5901

Closed
7 tasks done
sherifnada opened this issue Sep 8, 2021 · 0 comments
Closed
7 tasks done

[EPIC] SSH tunnel support for OLTP DB sources & destinations #5901

sherifnada opened this issue Sep 8, 2021 · 0 comments
Assignees
Labels
area/connectors Connector related issues lang/java lang/python priority/high High priority type/enhancement New feature or request

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Sep 8, 2021

Tell us about the problem you're trying to solve

This epic tracks adding support for connecting to OLTP databases via SSH tunnels.

Must-have DBs for Airbyte Cloud (in priority order)

  • Postgres source & Destination (already done)

Nice-to-have

Investigation of simulating an SSH bastion using Docker for running acceptance tests

Describe the solution you’d like

There's roughly two logical components here:

  1. Creating a blueprint for testing SSH connections using only Docker (currently Postgres is tested by creating an actual bastion and infrastructure). This step is optional but highly desirable. See the explanation here.
  2. Adding SSH support for each database.

Depending on whether you are adding a source or a destination the impl is slightly different. See the instructions for how to add SSH support to a connector

Example PR of adding SSH support to the Postgres source
Example of adding SSH support to Postgres destination

Note that both these PRs have way more logic in them than what you'll need to implement for a single connector because they add common helpers and shared libraries and docs

A note on implementation ordering

While testing via Docker (instead of creating actual DB instances on AWS) is highly desirable, it is optional. On the other hand, we must deliver SSH tunnels for the above DBs by the end of the month to support the Airbyte Cloud launch. My recommendation is to work on both pieces simultaneously to minimize delivery risk:

  1. Developers implementing SSH tunnels should do the fastest thing to unblock themselves. I suspect this means creating a DB in AWS through terraform or the UI. We already have a blueprint for how to create a database on AWS in terraform. You would need only to create a database without a public IP in the subnet which already exists, the same one the Postgres DB is currently linked to. Please talk to Oleksandra for getting access to the AWS dev environment where you can spin up a database. If you go this route, you can re-use many of the parameters (tunnel host, tunnel key, etc..) in the lastpass secrets Postgres Source ssh key creds and destination postgres test creds with sshkey. If Terraform is slowing you down too much feel free to create one in the UI.
  2. A different developer should work on investigating simulating an SSH bastion using Docker, as listed in this issue.

The complicating factor here is, if standing up infra in step #1 takes a long time then maybe it's a bad use of time if we can deliver #2 first. But this approach is just suggested to minimize risk as much as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues lang/java lang/python priority/high High priority type/enhancement New feature or request
Projects
No open projects
Archived in project
Development

No branches or pull requests

3 participants