🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) #6317

andriikorotkov · 2021-09-20T15:50:35Z

What

The goal of this PR is to add SSH only for the MySQL Destination. It does not try to deal with how it can be reused for other sources or destinations.
Took concept of how it was done for Postres

How

Handles injecting the SSH tunnel in the check, discover, read methods of MySQL Destination directly. This makes it relatively straightforward to inject without worrying about touching other jdbc dbs.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…5970-mysql-ssh-tunnel

…irbyte into akorotkov/5970-mysql-ssh-tunnel � Conflicts: � airbyte-integrations/bases/base-normalization/Dockerfile � airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java

…5970-mysql-ssh-tunnel

andriikorotkov · 2021-09-24T09:14:33Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269216345
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269216345
🐛

andriikorotkov · 2021-09-24T10:31:33Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269450421
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269450421
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     36    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    391    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     36    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    391    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     117      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     47    67%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    182    82%

andriikorotkov · 2021-09-24T11:14:07Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269577807
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1269577807

andriikorotkov · 2021-09-24T11:46:17Z

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1269671319
✅ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1269671319
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     36    74%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    391    62%

CLAassistant · 2021-09-27T17:25:02Z

All committers have signed the CLA.

airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java

...ration/java/io/airbyte/integrations/destination/mysql/SshMySQLDestinationAcceptanceTest.java

andriikorotkov · 2021-09-28T09:01:02Z

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1281853716
✅ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1281853716

…5970-mysql-ssh-tunnel

updated mysql tests

19dde68

andriikorotkov requested a review from VitaliiMaltsev September 20, 2021 15:50

github-actions bot added the area/connectors Connector related issues label Sep 20, 2021

andriikorotkov added 3 commits September 20, 2021 22:58

updated mysql tests

f48aef0

added mysql ssh tunnel tests by key

47795dc

fixed remarks

c4cb020

andriikorotkov requested a review from alexandr-shegeda September 21, 2021 15:51

andriikorotkov added 3 commits September 21, 2021 18:53

fixed remarks

3cc425b

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

3e6eecf

…5970-mysql-ssh-tunnel

updated DatabricksStreamCopier

a7d9820

github-actions bot added the normalization label Sep 22, 2021

andriikorotkov temporarily deployed to more-secrets September 22, 2021 12:24 Inactive

andriikorotkov and others added 5 commits September 22, 2021 22:01

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

8f9125c

…5970-mysql-ssh-tunnel

switch to custom file for ssh config in normalization

283f45d

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

eb61ee5

…5970-mysql-ssh-tunnel

Merge branch 'master' into george/normalisation-ssh-fix

cf2adfa

updated MySQL SSH tests

3a306c1

github-actions bot added the area/worker Related to worker label Sep 23, 2021

andriikorotkov temporarily deployed to more-secrets September 23, 2021 09:14 Inactive

Phlair and others added 3 commits September 23, 2021 10:56

bump version

82299ec

get local port properly

de3fc8e

andriikorotkov linked an issue Sep 23, 2021 that may be closed by this pull request

Add support & test cases for MySQL Destination via SSH tunnels #6316

Closed

andriikorotkov added 3 commits September 23, 2021 15:51

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

c16ddf9

…5970-mysql-ssh-tunnel

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

48eaa40

…5970-mysql-ssh-tunnel

updated assertSameValue for MySQL ssh tunnel

5b26cc5

github-actions bot removed the area/worker Related to worker label Sep 24, 2021

andriikorotkov temporarily deployed to more-secrets September 24, 2021 06:55 Inactive

updated image version and documentation

8b37401

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 24, 2021

jrhizor temporarily deployed to more-secrets September 24, 2021 08:44 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 09:16 Inactive

updated CI credentials

d4ef60e

alexandr-shegeda approved these changes Sep 24, 2021

View reviewed changes

andriikorotkov temporarily deployed to more-secrets September 24, 2021 10:04 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 10:34 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 11:15 Inactive

jrhizor temporarily deployed to more-secrets September 24, 2021 11:48 Inactive

andriikorotkov temporarily deployed to more-secrets September 24, 2021 12:12 Inactive

andriikorotkov requested a review from sherifnada September 24, 2021 12:54

andriikorotkov marked this pull request as ready for review September 24, 2021 12:55

andriikorotkov requested a review from tuliren September 27, 2021 09:40

sherifnada approved these changes Sep 27, 2021

View reviewed changes

airbyte-workers/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java Show resolved Hide resolved

...ration/java/io/airbyte/integrations/destination/mysql/SshMySQLDestinationAcceptanceTest.java Show resolved Hide resolved

updated normalization documentation

470e4b7

andriikorotkov temporarily deployed to more-secrets September 28, 2021 08:25 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 09:03 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

9b67131

…5970-mysql-ssh-tunnel

andriikorotkov temporarily deployed to more-secrets September 28, 2021 09:24 Inactive

andriikorotkov merged commit 8fa1571 into master Sep 28, 2021

andriikorotkov deleted the akorotkov/5970-mysql-ssh-tunnel branch September 28, 2021 10:11

This was referenced Sep 29, 2021

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6532

Closed

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6539

Closed

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6548

Merged

Phlair mentioned this pull request Oct 6, 2021

🎉Destination MySQl - added ssl strict encrypt connector #6763

Merged

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) #6317

🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) #6317

andriikorotkov commented Sep 20, 2021 •

edited

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

CLAassistant commented Sep 27, 2021 •

edited

Loading

andriikorotkov commented Sep 28, 2021 •

edited by github-actions bot

Loading

🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) #6317

🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) #6317

Conversation

andriikorotkov commented Sep 20, 2021 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

andriikorotkov commented Sep 24, 2021 • edited by github-actions bot Loading

andriikorotkov commented Sep 24, 2021 • edited by github-actions bot Loading

andriikorotkov commented Sep 24, 2021 • edited by github-actions bot Loading

andriikorotkov commented Sep 24, 2021 • edited by github-actions bot Loading

CLAassistant commented Sep 27, 2021 • edited Loading

andriikorotkov commented Sep 28, 2021 • edited by github-actions bot Loading

andriikorotkov commented Sep 20, 2021 •

edited

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Sep 24, 2021 •

edited by github-actions bot

Loading

CLAassistant commented Sep 27, 2021 •

edited

Loading

andriikorotkov commented Sep 28, 2021 •

edited by github-actions bot

Loading