Redshift Destination: Add SSH Tunnelling Config Option #23523

jcowanpdx · 2023-02-27T23:31:20Z

Acceptance tests pass and exercise both Insert and S3 type loading options as well as password and key based authentication with the SSH bastion host.

Airbyters: Credentials/configs for acceptance testing against AWS Redshift are updated in Google Cloud Secret Manager. Be sure to get both the config.json and config_staging.json as well as the new PEM file

What

User would like to be able to access their Redshift Destination database via an SSH Tunnelling config (SSH Bastion Host), similar to Postgres, MySQL, etc. destinations.

How

This change wraps the Destination constructor in the SSHWrappedDestination, which injects SSH configuration options into the connector configuration spec.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

We are not expecting any breaking changes as this is an enabling PR, adding new features and capabilities only.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Issue: #12131 User would like to be able to access their Redshift Destination database via an SSH Tunnelling config (SSH Bastion Host), just as in Postgres, etc. destinations. Acceptance tests pass and exercise both Insert and S3 type loading options as well as password and key based authentication with the SSH bastion host. Airbyters: Credentials/configs for acceptance testing against AWS Redshift are updated in Google Cloud Secret Manager. Be sure to get both the config.json and config_staging.json as well as the new PEM file

CLAassistant · 2023-02-27T23:31:27Z

All committers have signed the CLA.

github-actions · 2023-02-27T23:34:08Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (1)

Connector	Version	Changelog	Publish
`destination-redshift`	`0.4.0`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

Issue: #12131 User would like to be able to access their Redshift Destination database via an SSH Tunnelling config (SSH Bastion Host), just as in Postgres, etc. destinations. Acceptance tests pass and exercise both Insert and S3 type loading options as well as password and key based authentication with the SSH bastion host. Airbyters: Credentials/configs for acceptance testing against AWS Redshift are updated in Google Cloud Secret Manager. Be sure to get both the config.json and config_staging.json as well as the new PEM file

…com:airbytehq/airbyte into jeff/add-ssh-tunnel-to-redshift-destination

jcowanpdx · 2023-02-28T16:42:09Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4295284609
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4295284609
🐛 https://gradle.com/s/padauifeqyrps

Build Failed

Test summary info:

Could not find result summary

...a/io/airbyte/integrations/destination/redshift/SshRedshiftDestinationBaseAcceptanceTest.java

Update the version id. Touch up the docs.

jcowanpdx · 2023-02-28T21:08:17Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4297402723
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4297402723
🐛 https://gradle.com/s/bpcl3cqxudftu

Build Failed

Test summary info:

Could not find result summary

evantahler · 2023-02-28T21:41:33Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4297646524
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4297646524
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

evantahler

Conditionally approved assuming /test passes

airbyte-integrations/connectors/destination-redshift/Dockerfile

...a/io/airbyte/integrations/destination/redshift/SshRedshiftDestinationBaseAcceptanceTest.java

docs/integrations/destinations/redshift.md

edgao

one small question around the top-level destination class, otherwise lgtm

...redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftDestination.java

jcowanpdx · 2023-03-02T19:43:11Z

Confirmed that normalization will work through the tunnel configuration as well.

…, if provided.

…com:airbytehq/airbyte into jeff/add-ssh-tunnel-to-redshift-destination

jcowanpdx · 2023-03-03T21:45:03Z

/publish connector=connectors/destination-redshift

🕑 Publishing the following connectors:
connectors/destination-redshift
https://github.com/airbytehq/airbyte/actions/runs/4327523925

Connector	Did it publish?	Were definitions generated?
connectors/destination-redshift	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grishick · 2023-03-04T00:40:01Z

Looks like some integration tests are failing to connect to Redshift while executing publish command. This may be related to either our internal Redshift cluster being overwhelmed or the SSH bastion. At least one of the errors was this (looks more like the problem is with the SSH bastion):

        Caused by:
        java.sql.SQLTransientConnectionException: HikariPool-206 - Connection is not available, request timed out after 60005ms.
            at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
            at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
            at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
            at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
            at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83)
            ... 96 more

            Caused by:
            java.sql.SQLException: [Amazon](500310) Invalid operation: connection limit "500" exceeded for non-bootstrap users;
                at com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
                at com.amazon.redshift.client.InboundDataHandler.read(Unknown Source)
                at com.amazon.support.channels.AbstractSocketChannel.readCallback(Unknown Source)
                at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source)

                Caused by:
                com.amazon.support.exceptions.ErrorException: [Amazon](500310) Invalid operation: connection limit "500" exceeded for non-bootstrap users;
                    at app//com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
                    ... 3 more

jcowanpdx · 2023-03-06T16:45:15Z

Looks like some integration tests are failing to connect to Redshift while executing publish command. This may be related to either our internal Redshift cluster being overwhelmed or the SSH bastion. At least one of the errors was this (looks more like the problem is with the SSH bastion):

        Caused by:
        java.sql.SQLTransientConnectionException: HikariPool-206 - Connection is not available, request timed out after 60005ms.
            at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
            at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
            at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
            at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
            at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:83)
            ... 96 more

            Caused by:
            java.sql.SQLException: [Amazon](500310) Invalid operation: connection limit "500" exceeded for non-bootstrap users;
                at com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
                at com.amazon.redshift.client.InboundDataHandler.read(Unknown Source)
                at com.amazon.support.channels.AbstractSocketChannel.readCallback(Unknown Source)
                at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source)

                Caused by:
                com.amazon.support.exceptions.ErrorException: [Amazon](500310) Invalid operation: connection limit "500" exceeded for non-bootstrap users;
                    at app//com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
                    ... 3 more

seems more related to Redshift limits per some docs I'm reading. Raising the limit might not be a great solution as that requires some back and forth with Amazon. I suspect something is either leaking connections or we are running these tests in parallel at too high a rate. Not sure why integration tests are using so many connections per se.

jcowanpdx · 2023-03-06T21:18:10Z

/publish connector=connectors/destination-redshift

🕑 Publishing the following connectors:
connectors/destination-redshift
https://github.com/airbytehq/airbyte/actions/runs/4348047576

Connector	Did it publish?	Were definitions generated?
connectors/destination-redshift	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

jcowanpdx · 2023-03-07T00:11:54Z

The change to running integration tests in parallel, combined with a 4 hour idle session timeout in redshift causes an accumulation of idle sessions that cause resource shortages running tests. I have lowered the idle session time to 60 in Redshift and local testing look promising. Running another /publish run to verify

jcowanpdx · 2023-03-07T00:21:23Z

/publish connector=connectors/destination-redshift

🕑 Publishing the following connectors:
connectors/destination-redshift
https://github.com/airbytehq/airbyte/actions/runs/4349288708

Connector	Did it publish?	Were definitions generated?
connectors/destination-redshift	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

jcowanpdx · 2023-03-07T16:45:11Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4356472449
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4356472449
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

* Redshift Destination: Add SSH Tunnelling Config Option Issue: airbytehq#12131 User would like to be able to access their Redshift Destination database via an SSH Tunnelling config (SSH Bastion Host), just as in Postgres, etc. destinations. Acceptance tests pass and exercise both Insert and S3 type loading options as well as password and key based authentication with the SSH bastion host. Airbyters: Credentials/configs for acceptance testing against AWS Redshift are updated in Google Cloud Secret Manager. Be sure to get both the config.json and config_staging.json as well as the new PEM file * README.md updates ** Move the pem info into the secrets config files ** Update the version id to 0.4.0 ** Touch up the docs ** adding a note about S3 staging NOT using the SSH Tunnel configuration, if provided. ** make base class abstract to prevent double running the same tests Co-authored-by: Octavia Squidington III <[email protected]>

jcowanpdx requested a review from a team as a code owner February 27, 2023 23:31

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/redshift labels Feb 27, 2023

jcowanpdx linked an issue Feb 28, 2023 that may be closed by this pull request

Redshift Destination implement SSH tunneling #12131

Closed

jcowanpdx and others added 4 commits February 27, 2023 17:20

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

6cd1cf5

Merge branch 'jeff/add-ssh-tunnel-to-redshift-destination' of github.…

6463556

…com:airbytehq/airbyte into jeff/add-ssh-tunnel-to-redshift-destination

README.md updates

e7673ac

grishick reviewed Feb 28, 2023

View reviewed changes

...a/io/airbyte/integrations/destination/redshift/SshRedshiftDestinationBaseAcceptanceTest.java Outdated Show resolved Hide resolved

jbfbell approved these changes Feb 28, 2023

View reviewed changes

Move the pem info into the secrets config files.

6c82cbe

Update the version id. Touch up the docs.

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 28, 2023

evantahler approved these changes Feb 28, 2023

View reviewed changes

edgao approved these changes Feb 28, 2023

View reviewed changes

...redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftDestination.java Show resolved Hide resolved

jcowanpdx added 3 commits February 28, 2023 14:18

bump to 0.4.0

360e473

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

bbd94a7

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

c59f0aa

jcowanpdx added 5 commits March 2, 2023 11:46

adding a note about S3 staging NOT using the SSH Tunnel configuration…

0c933ec

…, if provided.

Merge branch 'jeff/add-ssh-tunnel-to-redshift-destination' of github.…

862ec5d

…com:airbytehq/airbyte into jeff/add-ssh-tunnel-to-redshift-destination

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

b684cd0

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

c527595

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

919408e

jcowanpdx added 2 commits March 6, 2023 11:05

limit idle connection time for redshift users

c915214

make base class abstract to prevent double running the same tests

dbaf59e

auto-bump connector version

acc8cee

jcowanpdx added 2 commits March 6, 2023 16:31

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

b436e01

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

b1addc6

Merge branch 'master' into jeff/add-ssh-tunnel-to-redshift-destination

2253606

jcowanpdx merged commit b6b4203 into master Mar 7, 2023

jcowanpdx deleted the jeff/add-ssh-tunnel-to-redshift-destination branch March 7, 2023 17:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redshift Destination: Add SSH Tunnelling Config Option #23523

Redshift Destination: Add SSH Tunnelling Config Option #23523

jcowanpdx commented Feb 27, 2023 •

edited

Loading

CLAassistant commented Feb 27, 2023 •

edited

Loading

github-actions bot commented Feb 27, 2023 •

edited

Loading

✅ Sources (0)

✅ Destinations (1)

👀 Other Modules (1)

Actionable Items

jcowanpdx commented Feb 28, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Feb 28, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 28, 2023 •

edited by github-actions bot

Loading

evantahler left a comment

edgao left a comment

jcowanpdx commented Mar 2, 2023

jcowanpdx commented Mar 3, 2023 •

edited by github-actions bot

Loading

grishick commented Mar 4, 2023 •

edited

Loading

jcowanpdx commented Mar 6, 2023

jcowanpdx commented Mar 6, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Mar 7, 2023

jcowanpdx commented Mar 7, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Mar 7, 2023 •

edited by github-actions bot

Loading

Redshift Destination: Add SSH Tunnelling Config Option #23523

Redshift Destination: Add SSH Tunnelling Config Option #23523

Conversation

jcowanpdx commented Feb 27, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Feb 27, 2023 • edited Loading

github-actions bot commented Feb 27, 2023 • edited Loading

Affected Connector Report

✅ Sources (0)

✅ Destinations (1)

👀 Other Modules (1)

Actionable Items

jcowanpdx commented Feb 28, 2023 • edited by github-actions bot Loading

Build Failed

jcowanpdx commented Feb 28, 2023 • edited by github-actions bot Loading

Build Failed

evantahler commented Feb 28, 2023 • edited by github-actions bot Loading

Build Passed

evantahler left a comment

Choose a reason for hiding this comment

edgao left a comment

Choose a reason for hiding this comment

jcowanpdx commented Mar 2, 2023

jcowanpdx commented Mar 3, 2023 • edited by github-actions bot Loading

grishick commented Mar 4, 2023 • edited Loading

jcowanpdx commented Mar 6, 2023

jcowanpdx commented Mar 6, 2023 • edited by github-actions bot Loading

jcowanpdx commented Mar 7, 2023

jcowanpdx commented Mar 7, 2023 • edited by github-actions bot Loading

jcowanpdx commented Mar 7, 2023 • edited by github-actions bot Loading

Build Passed

jcowanpdx commented Feb 27, 2023 •

edited

Loading

CLAassistant commented Feb 27, 2023 •

edited

Loading

github-actions bot commented Feb 27, 2023 •

edited

Loading

jcowanpdx commented Feb 28, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Feb 28, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 28, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Mar 3, 2023 •

edited by github-actions bot

Loading

grishick commented Mar 4, 2023 •

edited

Loading

jcowanpdx commented Mar 6, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Mar 7, 2023 •

edited by github-actions bot

Loading

jcowanpdx commented Mar 7, 2023 •

edited by github-actions bot

Loading