Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the dependency between the orchestrator container and the worker app #17570

Merged
merged 17 commits into from
Oct 12, 2022

Conversation

benmoriceau
Copy link
Contributor

@benmoriceau benmoriceau commented Oct 4, 2022

What

Remove the dependency from the orchestrator app to the worker app.

It is done by adding a new module airbyte-commons-worker. This module is containing the classes needed by the airbyte-container-orchestrator to the common module.

Recommended reading order

  1. build.gradle of airbyte-container-orchestrator -> remove the worker dependencies
  2. airbyte-commons-worker that contains a set of files that are move from the worker module to the commons worker module.

@github-actions github-actions bot added area/platform issues related to the platform area/worker Related to worker labels Oct 4, 2022
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-kinesis
  • destination-jdbc
  • destination-mongodb
  • destination-gcs
  • destination-aws-datalake
  • destination-bigquery
  • destination-scylla
  • destination-cassandra
  • destination-dev-null
  • destination-redis
  • destination-dynamodb
  • destination-mysql
  • destination-databricks
  • destination-mongodb-strict-encrypt
  • destination-clickhouse-strict-encrypt
  • destination-snowflake
  • destination-keen
  • destination-pubsub
  • destination-clickhouse
  • destination-mysql-strict-encrypt
  • destination-csv
  • destination-local-json
  • destination-mssql
  • destination-meilisearch
  • destination-oracle
  • destination-azure-blob-storage
  • destination-s3
  • destination-elasticsearch
  • destination-postgres-strict-encrypt
  • destination-mqtt
  • destination-redshift
  • destination-e2e-test
  • destination-oracle-strict-encrypt
  • destination-elasticsearch-strict-encrypt
  • destination-bigquery-denormalized
  • destination-tidb
  • destination-r2
  • destination-mariadb-columnstore
  • destination-pulsar
  • destination-mssql-strict-encrypt
  • destination-rockset
  • destination-kafka
  • destination-postgres

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 18:37 Inactive
@benmoriceau benmoriceau changed the title test [ci skip] Remove the dependency between the orchestrator container and the worker app Oct 10, 2022
@benmoriceau benmoriceau marked this pull request as draft October 10, 2022 18:57
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-redis
  • destination-e2e-test
  • destination-kafka
  • destination-s3
  • destination-mysql-strict-encrypt
  • destination-pubsub
  • destination-redshift
  • destination-mysql
  • destination-dev-null
  • destination-mqtt
  • destination-snowflake
  • destination-elasticsearch
  • destination-azure-blob-storage
  • destination-mssql-strict-encrypt
  • destination-rockset
  • destination-tidb
  • destination-mariadb-columnstore
  • destination-mongodb
  • destination-dynamodb
  • destination-mongodb-strict-encrypt
  • destination-csv
  • destination-scylla
  • destination-elasticsearch-strict-encrypt
  • destination-jdbc
  • destination-mssql
  • destination-aws-datalake
  • destination-meilisearch
  • destination-oracle-strict-encrypt
  • destination-cassandra
  • destination-bigquery
  • destination-kinesis
  • destination-postgres
  • destination-local-json
  • destination-postgres-strict-encrypt
  • destination-oracle
  • destination-bigquery-denormalized
  • destination-clickhouse
  • destination-r2
  • destination-gcs
  • destination-pulsar
  • destination-keen
  • destination-databricks
  • destination-clickhouse-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 20:03 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-bigquery
  • destination-keen
  • destination-dynamodb
  • destination-s3
  • destination-oracle-strict-encrypt
  • destination-mqtt
  • destination-databricks
  • destination-cassandra
  • destination-oracle
  • destination-mariadb-columnstore
  • destination-clickhouse
  • destination-kafka
  • destination-mysql-strict-encrypt
  • destination-pubsub
  • destination-clickhouse-strict-encrypt
  • destination-redshift
  • destination-azure-blob-storage
  • destination-csv
  • destination-local-json
  • destination-redis
  • destination-e2e-test
  • destination-scylla
  • destination-kinesis
  • destination-mssql-strict-encrypt
  • destination-mongodb-strict-encrypt
  • destination-mysql
  • destination-jdbc
  • destination-aws-datalake
  • destination-pulsar
  • destination-r2
  • destination-elasticsearch-strict-encrypt
  • destination-postgres-strict-encrypt
  • destination-rockset
  • destination-mssql
  • destination-snowflake
  • destination-elasticsearch
  • destination-postgres
  • destination-dev-null
  • destination-meilisearch
  • destination-tidb
  • destination-bigquery-denormalized
  • destination-gcs
  • destination-mongodb

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 20:49 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-elasticsearch-strict-encrypt
  • destination-pubsub
  • destination-mariadb-columnstore
  • destination-dynamodb
  • destination-oracle-strict-encrypt
  • destination-mysql
  • destination-bigquery
  • destination-mysql-strict-encrypt
  • destination-mongodb-strict-encrypt
  • destination-clickhouse
  • destination-keen
  • destination-r2
  • destination-scylla
  • destination-postgres-strict-encrypt
  • destination-s3
  • destination-meilisearch
  • destination-redshift
  • destination-mssql-strict-encrypt
  • destination-rockset
  • destination-mqtt
  • destination-databricks
  • destination-aws-datalake
  • destination-mssql
  • destination-clickhouse-strict-encrypt
  • destination-jdbc
  • destination-local-json
  • destination-cassandra
  • destination-dev-null
  • destination-csv
  • destination-mongodb
  • destination-azure-blob-storage
  • destination-bigquery-denormalized
  • destination-oracle
  • destination-e2e-test
  • destination-redis
  • destination-postgres
  • destination-kafka
  • destination-kinesis
  • destination-pulsar
  • destination-elasticsearch
  • destination-snowflake
  • destination-gcs
  • destination-tidb

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 20:57 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-pubsub
  • destination-tidb
  • destination-cassandra
  • destination-clickhouse-strict-encrypt
  • destination-e2e-test
  • destination-elasticsearch
  • destination-local-json
  • destination-mssql
  • destination-snowflake
  • destination-dev-null
  • destination-bigquery
  • destination-azure-blob-storage
  • destination-kafka
  • destination-oracle
  • destination-bigquery-denormalized
  • destination-databricks
  • destination-aws-datalake
  • destination-oracle-strict-encrypt
  • destination-s3
  • destination-csv
  • destination-gcs
  • destination-dynamodb
  • destination-mysql-strict-encrypt
  • destination-redis
  • destination-keen
  • destination-mariadb-columnstore
  • destination-mssql-strict-encrypt
  • destination-r2
  • destination-postgres
  • destination-postgres-strict-encrypt
  • destination-kinesis
  • destination-clickhouse
  • destination-mysql
  • destination-mqtt
  • destination-mongodb
  • destination-pulsar
  • destination-meilisearch
  • destination-elasticsearch-strict-encrypt
  • destination-redshift
  • destination-mongodb-strict-encrypt
  • destination-rockset
  • destination-scylla
  • destination-jdbc

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 21:44 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-elasticsearch-strict-encrypt
  • destination-mariadb-columnstore
  • source-alloydb
  • source-oracle
  • destination-mongodb-strict-encrypt
  • source-cockroachdb-strict-encrypt
  • source-tidb
  • source-clickhouse
  • source-e2e-test
  • source-clickhouse-strict-encrypt
  • destination-snowflake
  • destination-pulsar
  • source-alloydb-strict-encrypt
  • source-elasticsearch
  • destination-oracle
  • destination-bigquery
  • destination-r2
  • destination-mssql-strict-encrypt
  • destination-oracle-strict-encrypt
  • source-mysql
  • destination-meilisearch
  • destination-azure-blob-storage
  • destination-gcs
  • destination-postgres
  • destination-aws-datalake
  • source-mssql
  • destination-bigquery-denormalized
  • source-db2
  • destination-keen
  • destination-scylla
  • destination-elasticsearch
  • destination-jdbc
  • destination-clickhouse-strict-encrypt
  • destination-kafka
  • destination-kinesis
  • source-cockroachdb
  • source-mysql-strict-encrypt
  • destination-cassandra
  • source-postgres-strict-encrypt
  • destination-dev-null
  • source-bigquery
  • destination-mysql
  • source-oracle-strict-encrypt
  • source-scaffold-java-jdbc
  • source-postgres
  • source-kafka
  • destination-mqtt
  • destination-redshift
  • destination-redis
  • destination-databricks
  • destination-s3
  • source-snowflake
  • source-jdbc
  • source-sftp
  • source-mssql-strict-encrypt
  • destination-tidb
  • destination-mssql
  • destination-csv
  • destination-rockset
  • destination-e2e-test
  • destination-mysql-strict-encrypt
  • source-db2-strict-encrypt
  • source-redshift
  • destination-mongodb
  • destination-clickhouse
  • source-mongodb-strict-encrypt
  • source-e2e-test-cloud
  • destination-dynamodb
  • destination-postgres-strict-encrypt
  • destination-pubsub
  • destination-local-json
  • source-mongodb-v2

@benmoriceau benmoriceau temporarily deployed to more-secrets October 10, 2022 21:51 Inactive
@benmoriceau benmoriceau marked this pull request as ready for review October 10, 2022 22:24
@benmoriceau
Copy link
Contributor Author

@colesnodgrass @alovew The build should pass so it is ready to review.

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-azure-blob-storage
  • source-bigquery
  • destination-pubsub
  • source-alloydb-strict-encrypt
  • source-mysql
  • source-alloydb
  • destination-tidb
  • source-mssql-strict-encrypt
  • source-clickhouse-strict-encrypt
  • destination-mysql-strict-encrypt
  • destination-scylla
  • destination-cassandra
  • source-postgres
  • destination-mssql
  • source-clickhouse
  • source-mysql-strict-encrypt
  • source-mongodb-strict-encrypt
  • destination-mqtt
  • destination-rockset
  • source-jdbc
  • source-db2
  • destination-pulsar
  • destination-kinesis
  • source-tidb
  • destination-meilisearch
  • destination-mariadb-columnstore
  • destination-mongodb-strict-encrypt
  • source-e2e-test-cloud
  • destination-dynamodb
  • source-e2e-test
  • source-cockroachdb-strict-encrypt
  • destination-local-json
  • destination-postgres
  • source-elasticsearch
  • destination-csv
  • source-postgres-strict-encrypt
  • destination-aws-datalake
  • destination-mongodb
  • destination-mysql
  • destination-e2e-test
  • source-mssql
  • destination-redshift
  • destination-oracle
  • destination-gcs
  • destination-clickhouse
  • destination-clickhouse-strict-encrypt
  • source-cockroachdb
  • destination-bigquery
  • destination-bigquery-denormalized
  • destination-elasticsearch-strict-encrypt
  • destination-redis
  • source-kafka
  • destination-kafka
  • source-oracle-strict-encrypt
  • source-redshift
  • destination-snowflake
  • source-oracle
  • source-scaffold-java-jdbc
  • destination-r2
  • destination-dev-null
  • destination-mssql-strict-encrypt
  • source-mongodb-v2
  • destination-databricks
  • destination-oracle-strict-encrypt
  • destination-s3
  • source-snowflake
  • source-db2-strict-encrypt
  • destination-jdbc
  • destination-postgres-strict-encrypt
  • destination-elasticsearch
  • source-sftp
  • destination-keen

@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 15:25 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-redis
  • destination-local-json
  • destination-s3
  • source-db2
  • source-snowflake
  • source-db2-strict-encrypt
  • destination-r2
  • destination-cassandra
  • source-e2e-test-cloud
  • destination-clickhouse-strict-encrypt
  • destination-bigquery-denormalized
  • destination-elasticsearch
  • destination-bigquery
  • destination-postgres-strict-encrypt
  • destination-postgres
  • destination-mqtt
  • source-clickhouse
  • destination-kafka
  • destination-mongodb
  • destination-pulsar
  • destination-rockset
  • destination-azure-blob-storage
  • source-mongodb-strict-encrypt
  • source-kafka
  • source-mysql
  • source-oracle-strict-encrypt
  • source-tidb
  • destination-dev-null
  • source-bigquery
  • destination-csv
  • destination-redshift
  • source-oracle
  • destination-e2e-test
  • destination-mssql
  • destination-pubsub
  • source-alloydb
  • destination-tidb
  • destination-snowflake
  • source-postgres
  • source-clickhouse-strict-encrypt
  • source-mssql-strict-encrypt
  • destination-aws-datalake
  • source-mysql-strict-encrypt
  • source-mongodb-v2
  • source-elasticsearch
  • destination-oracle
  • source-sftp
  • destination-mongodb-strict-encrypt
  • destination-databricks
  • destination-oracle-strict-encrypt
  • destination-gcs
  • destination-keen
  • source-cockroachdb
  • source-jdbc
  • source-mssql
  • destination-meilisearch
  • destination-kinesis
  • destination-elasticsearch-strict-encrypt
  • destination-mysql-strict-encrypt
  • destination-dynamodb
  • destination-jdbc
  • destination-scylla
  • destination-clickhouse
  • destination-mssql-strict-encrypt
  • source-scaffold-java-jdbc
  • destination-mariadb-columnstore
  • source-e2e-test
  • source-alloydb-strict-encrypt
  • source-redshift
  • destination-mysql
  • source-cockroachdb-strict-encrypt
  • source-postgres-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 15:34 Inactive
@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 16:01 Inactive
@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 16:54 Inactive
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@benmoriceau
Copy link
Contributor Author

@colesnodgrass @jdpgrailsdev I have added 628b721 to fix the acceptance test.

@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 21:25 Inactive
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@benmoriceau benmoriceau requested a review from a team as a code owner October 11, 2022 22:20
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-bigquery-denormalized
  • destination-oracle
  • source-postgres
  • destination-e2e-test
  • destination-mysql
  • source-e2e-test
  • source-alloydb
  • destination-azure-blob-storage
  • destination-kinesis
  • destination-mysql-strict-encrypt
  • source-alloydb-strict-encrypt
  • destination-elasticsearch
  • destination-jdbc
  • source-sftp
  • destination-redis
  • destination-local-json
  • source-mongodb-v2
  • source-tidb
  • source-e2e-test-cloud
  • source-jdbc
  • source-cockroachdb-strict-encrypt
  • destination-pubsub
  • destination-snowflake
  • destination-meilisearch
  • destination-keen
  • source-snowflake
  • destination-rockset
  • destination-mongodb-strict-encrypt
  • source-cockroachdb
  • destination-elasticsearch-strict-encrypt
  • source-scaffold-java-jdbc
  • destination-redshift
  • destination-mongodb
  • destination-kafka
  • source-redshift
  • destination-dynamodb
  • destination-postgres-strict-encrypt
  • source-mysql
  • destination-aws-datalake
  • destination-tidb
  • destination-mariadb-columnstore
  • destination-databricks
  • source-elasticsearch
  • source-postgres-strict-encrypt
  • destination-scylla
  • source-clickhouse
  • source-mysql-strict-encrypt
  • source-db2
  • source-oracle-strict-encrypt
  • destination-postgres
  • source-oracle
  • destination-gcs
  • destination-clickhouse
  • destination-mqtt
  • destination-dev-null
  • destination-r2
  • destination-csv
  • source-kafka
  • source-clickhouse-strict-encrypt
  • destination-mssql-strict-encrypt
  • source-mongodb-strict-encrypt
  • source-mssql
  • destination-oracle-strict-encrypt
  • destination-cassandra
  • source-db2-strict-encrypt
  • destination-pulsar
  • destination-bigquery
  • source-mssql-strict-encrypt
  • destination-mssql
  • destination-s3
  • source-bigquery
  • destination-clickhouse-strict-encrypt

@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 22:22 Inactive
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • destination-mongodb-strict-encrypt
  • destination-clickhouse
  • source-db2
  • source-scaffold-java-jdbc
  • destination-s3
  • destination-oracle-strict-encrypt
  • source-cockroachdb
  • source-cockroachdb-strict-encrypt
  • source-mysql
  • destination-redis
  • destination-kinesis
  • destination-bigquery
  • destination-tidb
  • destination-meilisearch
  • destination-r2
  • destination-local-json
  • destination-csv
  • source-e2e-test
  • destination-elasticsearch-strict-encrypt
  • destination-mysql-strict-encrypt
  • source-sftp
  • destination-azure-blob-storage
  • source-mssql-strict-encrypt
  • source-jdbc
  • destination-pulsar
  • destination-mqtt
  • source-e2e-test-cloud
  • source-postgres
  • destination-mssql-strict-encrypt
  • source-mongodb-v2
  • destination-cassandra
  • destination-aws-datalake
  • destination-elasticsearch
  • destination-mysql
  • destination-kafka
  • destination-e2e-test
  • source-db2-strict-encrypt
  • destination-dynamodb
  • source-postgres-strict-encrypt
  • source-elasticsearch
  • destination-mongodb
  • source-mysql-strict-encrypt
  • destination-dev-null
  • destination-rockset
  • source-clickhouse-strict-encrypt
  • source-alloydb-strict-encrypt
  • source-kafka
  • source-mssql
  • destination-databricks
  • destination-redshift
  • source-tidb
  • destination-postgres
  • source-alloydb
  • destination-jdbc
  • source-oracle
  • destination-oracle
  • destination-postgres-strict-encrypt
  • source-mongodb-strict-encrypt
  • destination-pubsub
  • source-bigquery
  • destination-snowflake
  • source-clickhouse
  • destination-gcs
  • destination-clickhouse-strict-encrypt
  • destination-bigquery-denormalized
  • destination-keen
  • destination-mssql
  • destination-mariadb-columnstore
  • source-snowflake
  • source-oracle-strict-encrypt
  • source-redshift
  • destination-scylla

@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 22:46 Inactive
@benmoriceau benmoriceau temporarily deployed to more-secrets October 11, 2022 23:29 Inactive
@benmoriceau benmoriceau merged commit f650005 into master Oct 12, 2022
@benmoriceau benmoriceau deleted the bmoric/remove-dep-container-orchestrator-worker branch October 12, 2022 00:12
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
…er app (airbytehq#17570)

* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* Add missing dependencies

* PR comments

* Fix acceptance test and add the seed dependency

* Fix build

* Fix build again
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/platform issues related to the platform area/server area/worker Related to worker connectors/destination/gcs normalization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants