Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) #7327

Merged
merged 16 commits into from
Oct 26, 2021
Merged

🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) #7327

merged 16 commits into from
Oct 26, 2021

Conversation

etsybaev
Copy link
Contributor

@etsybaev etsybaev commented Oct 25, 2021 •
edited
Loading

What

The was no way to connect to Clickhouse via ssh tunnel.

How

Added an option to connect to DB via SSH tunnel (aka Bastion server).
To achieve it there is a common approach with SshWrappedSource decorator had been used (the same way like we did with other connectors).

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here
  • Connector strict-encrypt version bumped and published

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

@etsybaev
[Ticket 6435] Source-ClickHouse: added ssl support
68a8f78
@etsybaev
[Ticket 6716] Source-ClickHouse: added secure only connector
80373dd
@etsybaev
[Ticket 6435] Source-ClickHouse: added ssl support
bf97f7b
@etsybaev
Updated checkstyle
a39308f
@etsybaev
Merge branch 'master' into etsybaev/6435-source-clickhouse-added-ssl-…
f4e592f 
…support

# Conflicts:
#	airbyte-integrations/connectors/source-clickhouse/src/main/java/io/airbyte/integrations/source/clickhouse/ClickHouseSource.java
#	airbyte-integrations/connectors/source-clickhouse/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/ClickHouseSourceAcceptanceTest.java
@etsybaev
Fixed merge conflict, updated docs and fix typo in module's name
70c0636
@etsybaev
updated docs
d897d6d
@etsybaev
Merge branch 'master' into etsybaev/6435-source-clickhouse-added-ssl-…
0c896f8 
…support
@etsybaev
Fixed tests as per comments in PR
fd4626f
@etsybaev
[7213] Source-Clickhouse: added connection via ssh support
9dd1b18
@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 25, 2021
@etsybaev etsybaev linked an issue Oct 25, 2021 that may be closed by this pull request
🎉 Source Clickhouse: add SSH tunnel support #7213
Closed
@github-actions github-actions bot removed the area/documentation Improvements or additions to documentation label Oct 25, 2021
@etsybaev etsybaev temporarily deployed to more-secrets October 25, 2021 12:17 Inactive
@etsybaev etsybaev temporarily deployed to more-secrets October 25, 2021 13:14 Inactive
@etsybaev etsybaev temporarily deployed to more-secrets October 25, 2021 13:53 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 25, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-clickhouse

🕑 connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1381659133
✅ connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1381659133
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 25, 2021 15:03 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 25, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-clickhouse-strict-encrypt

🕑 connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1381773140
✅ connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1381773140
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 25, 2021 15:28 Inactive
@etsybaev etsybaev changed the title [DRAFT_PR_IGNORE_IT] Etsybaev/7213 source clickhouse added ssh support [DRAFT_PR_IGNORE_IT] 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) Oct 26, 2021
@alexandr-shegeda
Copy link
Contributor

@etsybaev kindly remind: update required documentation

@etsybaev etsybaev temporarily deployed to more-secrets October 26, 2021 09:50 Inactive
@etsybaev
Copy link
Contributor Author

@etsybaev kindly remind: update required documentation

Sure. I will update it along with the bumping version once got all approvals. Thanks

@etsybaev etsybaev changed the title [DRAFT_PR_IGNORE_IT] 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) [DRAFT_PR] 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) Oct 26, 2021
@etsybaev etsybaev changed the title [DRAFT_PR] 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) 🎉Source Clickhouse: added option to connect via SSH tunnel (aka Bastion server) Oct 26, 2021
@etsybaev etsybaev marked this pull request as ready for review October 26, 2021 13:53
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Oct 26, 2021
@etsybaev etsybaev temporarily deployed to more-secrets October 26, 2021 16:13 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 26, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-clickhouse

🕑 connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386636930
✅ connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386636930
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 26, 2021 16:53 Inactive
@etsybaev
Merge branch 'master' into etsybaev/7213-source-clickhouse-added-ssh-…
a532993 
…support

# Conflicts:
#	airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@etsybaev
fixed merge conflict
eff7b5c
@etsybaev etsybaev temporarily deployed to more-secrets October 26, 2021 17:02 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 26, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-clickhouse

🕑 connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386829453
✅ connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386829453
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 26, 2021 17:50 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 26, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-clickhouse-strict-encrypt

🕑 connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1386893035
✅ connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1386893035
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 26, 2021 18:09 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 26, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-clickhouse

🕑 connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386921708
✅ connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1386921708

@jrhizor jrhizor temporarily deployed to more-secrets October 26, 2021 18:16 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Oct 26, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-clickhouse-strict-encrypt

🕑 connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1386963410
✅ connectors/source-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1386963410

@jrhizor jrhizor temporarily deployed to more-secrets October 26, 2021 18:28 Inactive
@etsybaev etsybaev merged commit 46a249e into master Oct 26, 2021
@etsybaev etsybaev deleted the etsybaev/7213-source-clickhouse-added-ssh-support branch October 26, 2021 18:39
@jrhizor jrhizor mentioned this pull request Nov 1, 2021
Merged
@mkhokh-33 mkhokh-33 mentioned this pull request Nov 5, 2021
Merged
38 tasks
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
@etsybaev @schlattk
🎉Source Clickhouse: added option to connect via SSH tunnel (aka Basti…
c768e9e 
…on server) (airbytehq#7327)

Source-Clickhouse: added support for connection via ssh tunnel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sherifnada sherifnada sherifnada approved these changes

@alexandr-shegeda alexandr-shegeda alexandr-shegeda approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/clickhouse connectors/sources-database
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

🎉 Source Clickhouse: add SSH tunnel support
5 participants
@etsybaev @alexandr-shegeda @sherifnada @jrhizor @igrankova