Bmoric/remove dep server worker #17894

benmoriceau · 2022-10-12T15:52:48Z

What

Remove the dependency between server and worker

Recommended reading order

airbyte-worker-models -> new module that contains the worker models.
TemporalClient.java -> the entire TemporalClient has been move to the common package
ConnectionManagerUtils.java -> the entire ConnectionManagerUtils has been move to the common package
airbyte-server/build.gradle -> remove the dependency on the worker

…orker

…ove-dep-container-orchestrator-worker

…hub.com:airbytehq/airbyte into bmoric/remove-dep-server-worker

…ove-dep-server-worker

github-actions · 2022-10-12T15:53:36Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-clickhouse
destination-bigquery
destination-redshift
destination-mysql
destination-mysql-strict-encrypt
destination-jdbc
destination-tidb
destination-mssql
destination-clickhouse-strict-encrypt
destination-postgres
destination-bigquery-denormalized
destination-oracle
destination-oracle-strict-encrypt
destination-postgres-strict-encrypt
destination-snowflake
destination-mssql-strict-encrypt

…q/airbyte into bmoric/remove-dep-server-worker

jdpgrailsdev · 2022-10-17T13:48:47Z

airbyte-worker-models/build.gradle

+    id "java-library"
+    id 'com.github.eirnym.js2p' version '1.0'
+}
+dependencies {


Nit: you can remove this if there are no additional dependencies required. That being said, if the generated code has dependencies on third party libraries, we should declare them here (even if they are added in the base build.gradle) so it is documented.

Fasterxml is added to the main build.gradle, I have removed the block

jdpgrailsdev · 2022-10-17T13:49:49Z

kube/resources/cron.yaml

+              valueFrom:
+                configMapKeyRef:
+                  name: airbyte-env
+                  key: WORKERS_MICRONAUT_ENVIRONMENTS


We should have a different name for the Cron environments in case it ends up having different environments than workers.

ok, I will do it. I am wondering how well this will work on cloud.

jdpgrailsdev · 2022-10-17T13:52:01Z

@benmoriceau Looks like the build failed, but I can't seem to figure out why it failed. Might have been a transient CI issue?

benmoriceau · 2022-10-17T15:22:18Z

@benmoriceau Looks like the build failed, but I can't seem to figure out why it failed. Might have been a transient CI issue?

It failed because OctaviaCli failed during the docker image build. IMO it is transient. I have rebase master on the branch. Hopefully this will be green now.

* test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt

* Implement ColumnSortButton component * Updates component name; Moves component to ui/Table folder; Refactors formattedMessageId property into using render content as children directly; Removes minor SortIcon component * Update airbyte-webapp/src/App.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Updates next properties: wasActive -> isActive, lowToLarge -> isAscending * Skip psql stop in acceptance test for gke (#18023) * Checks for iterator hasNext element (#18041) * Checks for iterator hasNext element * Fix linter with newline * Add Message Migration to Destination Connection Checks (#17954) * Add Message Migration to Destination Connection Checks * Fix test setup * Update helm release workflow (#18048) * Update workflow * Update trigger rules * fix: Update release workflow with abillity to add tags * Update workflow * Remove unused `airbyte-cli` (#18009) * 🐛 [low-code] $options shouldn't overwrite values that are already defined (#18060) * fix * Add missing test * remove prints * extract to method * rename * Add missing test * rename * bump * Update helm chart comments (#18072) * Update helm charts (#18073) * add test * fix chart.yaml * 16250 Destination Redis: Add SSH support (#17951) * 16250 Destination Redis: Add SSH support * 16250 Resolve port issue * 11679 Bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump helm chart version reference to 0.40.20 (#18074) * Bump helm chart version reference to 0.40.20 * remove binary Co-authored-by: xpuska513 <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * Helm Chart: Create service annotations for airbyte-server (#17932) * Support annotations for airbyte-server as well, update version and update docs. * Fix auto-indent. Co-authored-by: Kyryl Skobylko <[email protected]> * Bmoric/remove dep server worker (#17894) * test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt * Updated connector catalog page (#18076) * Move the port forward outside of the main docker-compose (#17864) * Bump Airbyte version from 0.40.14 to 0.40.15 (#17970) Co-authored-by: benmoriceau <[email protected]> * 🎉 Source Shopify: Add metafield streams (#17962) * 🎉 Source Shopify: Add metafield streams * Source Shopify: fix unittest * Source Shopify: docs update * Source Shopify: fix backward compatibility test * Source Shopify: fix schemas * Source Shopify: fix state filter * Source Shopify: refactor & optimize * Source Shopify: fix test privileges * Source Shopify: fix stream filter * Source Shopify: fix streams * Source Shopify: update abnormal state * Source Shopify: fix abnormal state streams * Source Shopify: fix streams * updated methods, formated code * Source Shopify: typo fix * auto-bump connector version Co-authored-by: Oleksandr Bazarnov <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * fix check for streams that do not use a stream slicer (#18080) * fix check for streams that do not use a stream slicer * increment version and changelog before publish * tolerate database nulls in webhook operation configs (#18084) * Implement webhook operation in the sync workflow (#18022) Implements the webhook operation as part of the sync workflow. - Introduces the new activity implementation - Updates the various interfaces that pass input to get the relevant configs to the sync workflow - Hooks the new activity into the sync workflow - Passes the webhook configs along into the sync workflow job * Bump helm chart version reference to 0.40.22 (#18077) * Added new "filters" python file, along with a "hash" filter. This can… (#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <[email protected]> * Bump helm chart version reference to 0.40.24 (#18081) * Bump helm chart version reference to 0.40.24 * Update .gitignore Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * SATs: allow new records in a sequential read for full refresh test (#17660) * SATs: allow new records in a sequential read for full refresh test * SATs: upd changelog * SATs: change the output when failing full refresh test * SATs: upd according to code review * Source facebook-marketing: remove `pixel` from custom conversions stream (#18045) * #744 source facebook-marketing: rm pixel from custom conversions stream * #744 source fb marketing: upd changelog * #744 source facebook-marketing - add custom_conversions to the test catalog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * #17506 fix klaviyo & marketo expected_records (#18101) Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Ryan Fu <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: Yevhen Sukhomud <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: xpuska513 <[email protected]> Co-authored-by: Prasanth <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: Amruta Ranade <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Artem Inzhyyants <[email protected]> Co-authored-by: Oleksandr Bazarnov <[email protected]> Co-authored-by: Brian Lai <[email protected]> Co-authored-by: Michael Siega <[email protected]> Co-authored-by: Alexander Marquardt <[email protected]> Co-authored-by: Denys Davydov <[email protected]>

* test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt

* Implement ColumnSortButton component * Updates component name; Moves component to ui/Table folder; Refactors formattedMessageId property into using render content as children directly; Removes minor SortIcon component * Update airbyte-webapp/src/App.tsx Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> * Updates next properties: wasActive -> isActive, lowToLarge -> isAscending * Skip psql stop in acceptance test for gke (airbytehq#18023) * Checks for iterator hasNext element (airbytehq#18041) * Checks for iterator hasNext element * Fix linter with newline * Add Message Migration to Destination Connection Checks (airbytehq#17954) * Add Message Migration to Destination Connection Checks * Fix test setup * Update helm release workflow (airbytehq#18048) * Update workflow * Update trigger rules * fix: Update release workflow with abillity to add tags * Update workflow * Remove unused `airbyte-cli` (airbytehq#18009) * 🐛 [low-code] $options shouldn't overwrite values that are already defined (airbytehq#18060) * fix * Add missing test * remove prints * extract to method * rename * Add missing test * rename * bump * Update helm chart comments (airbytehq#18072) * Update helm charts (airbytehq#18073) * add test * fix chart.yaml * 16250 Destination Redis: Add SSH support (airbytehq#17951) * 16250 Destination Redis: Add SSH support * 16250 Resolve port issue * 11679 Bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Bump helm chart version reference to 0.40.20 (airbytehq#18074) * Bump helm chart version reference to 0.40.20 * remove binary Co-authored-by: xpuska513 <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * Helm Chart: Create service annotations for airbyte-server (airbytehq#17932) * Support annotations for airbyte-server as well, update version and update docs. * Fix auto-indent. Co-authored-by: Kyryl Skobylko <[email protected]> * Bmoric/remove dep server worker (airbytehq#17894) * test [ci skip] * Autogenerated files * Add missing annotation * Remove unused json2Schema block from worker * Move tess * Missing deps and format * Fix test build * TMP * Add missing dependencies * PR comments * Tmp * [ci skip] Tmp * Fix acceptance test and add the seed dependency * Fix build * For diff * tmp * Build pass * make the worker to be on the platform only * fix setting.yaml * Fix pmd * Fix Cron * Add chart * Fix cron * Fix server build.gradle * Fix jar conflict * PR comments * Add cron micronaut environemnt * Updated connector catalog page (airbytehq#18076) * Move the port forward outside of the main docker-compose (airbytehq#17864) * Bump Airbyte version from 0.40.14 to 0.40.15 (airbytehq#17970) Co-authored-by: benmoriceau <[email protected]> * 🎉 Source Shopify: Add metafield streams (airbytehq#17962) * 🎉 Source Shopify: Add metafield streams * Source Shopify: fix unittest * Source Shopify: docs update * Source Shopify: fix backward compatibility test * Source Shopify: fix schemas * Source Shopify: fix state filter * Source Shopify: refactor & optimize * Source Shopify: fix test privileges * Source Shopify: fix stream filter * Source Shopify: fix streams * Source Shopify: update abnormal state * Source Shopify: fix abnormal state streams * Source Shopify: fix streams * updated methods, formated code * Source Shopify: typo fix * auto-bump connector version Co-authored-by: Oleksandr Bazarnov <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * fix check for streams that do not use a stream slicer (airbytehq#18080) * fix check for streams that do not use a stream slicer * increment version and changelog before publish * tolerate database nulls in webhook operation configs (airbytehq#18084) * Implement webhook operation in the sync workflow (airbytehq#18022) Implements the webhook operation as part of the sync workflow. - Introduces the new activity implementation - Updates the various interfaces that pass input to get the relevant configs to the sync workflow - Hooks the new activity into the sync workflow - Passes the webhook configs along into the sync workflow job * Bump helm chart version reference to 0.40.22 (airbytehq#18077) * Added new "filters" python file, along with a "hash" filter. This can… (airbytehq#18000) * Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future. * Added additional comments * Moved usage of the hash_obj inside the conditional that confirms it exists * Moved the hash function call inside a condition to ensure that it exists * Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in. * Added unit tests to validate new jinja hash functionality * Updated unit test to pass numeric value as a float instead of string * Removed unreferenced import to pytest * Updated version * format * format * format * format * format Co-authored-by: Alexandre Girard <[email protected]> * Bump helm chart version reference to 0.40.24 (airbytehq#18081) * Bump helm chart version reference to 0.40.24 * Update .gitignore Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * SATs: allow new records in a sequential read for full refresh test (airbytehq#17660) * SATs: allow new records in a sequential read for full refresh test * SATs: upd changelog * SATs: change the output when failing full refresh test * SATs: upd according to code review * Source facebook-marketing: remove `pixel` from custom conversions stream (airbytehq#18045) * airbytehq#744 source facebook-marketing: rm pixel from custom conversions stream * airbytehq#744 source fb marketing: upd changelog * airbytehq#744 source facebook-marketing - add custom_conversions to the test catalog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * #17506 fix klaviyo & marketo expected_records (airbytehq#18101) Co-authored-by: Edmundo Ruiz Ghanem <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Ryan Fu <[email protected]> Co-authored-by: Jimmy Ma <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: Yevhen Sukhomud <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: xpuska513 <[email protected]> Co-authored-by: Prasanth <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: Amruta Ranade <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: benmoriceau <[email protected]> Co-authored-by: Artem Inzhyyants <[email protected]> Co-authored-by: Oleksandr Bazarnov <[email protected]> Co-authored-by: Brian Lai <[email protected]> Co-authored-by: Michael Siega <[email protected]> Co-authored-by: Alexander Marquardt <[email protected]> Co-authored-by: Denys Davydov <[email protected]>

benmoriceau added 22 commits October 4, 2022 10:20

test [ci skip]

33b83bf

Remove the dependency between orchestrator APP and the worker

4134048

Merge branch 'master' into bmoric/remove-dep-container-orchestrator-w…

0d20286

…orker

Autogenerated files

d37bc19

Add missing annotation

bcf3431

Remove unused json2Schema block from worker

db1f95d

Move tess

e7d1592

Missing deps and format

0881cee

Fix test build

5c74045

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/rem…

7e2d449

…ove-dep-container-orchestrator-worker

TMP

203071a

Add missing dependencies

390b8eb

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/rem…

4b6634d

…ove-dep-container-orchestrator-worker

PR comments

cbae067

Tmp

7efa2c7

[ci skip] Tmp

636e507

Fix acceptance test and add the seed dependency

628b721

Fix build

4c0de3c

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/rem…

a3bce17

…ove-dep-container-orchestrator-worker

Merge branch 'bmoric/remove-dep-container-orchestrator-worker' of git…

279447f

…hub.com:airbytehq/airbyte into bmoric/remove-dep-server-worker

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/rem…

b1defe5

…ove-dep-server-worker

For diff

4b2c8f2

github-actions bot added area/platform issues related to the platform area/server area/worker Related to worker normalization labels Oct 12, 2022

tmp

72a11f8

github-actions bot removed the normalization label Oct 12, 2022

benmoriceau temporarily deployed to more-secrets October 12, 2022 16:37 Inactive

benmoriceau temporarily deployed to more-secrets October 14, 2022 15:32 Inactive

Merge branch 'master' into bmoric/remove-dep-server-worker

7f915cb

benmoriceau temporarily deployed to more-secrets October 14, 2022 22:05 Inactive

benmoriceau added 3 commits October 14, 2022 16:06

Fix server build.gradle

d241c64

Fix jar conflict

fada02f

Merge branch 'bmoric/remove-dep-server-worker' of github.com:airbyteh…

2f12031

…q/airbyte into bmoric/remove-dep-server-worker

benmoriceau temporarily deployed to more-secrets October 14, 2022 23:26 Inactive

jdpgrailsdev reviewed Oct 17, 2022

View reviewed changes

Merge branch 'master' into bmoric/remove-dep-server-worker

5d0cac0

benmoriceau temporarily deployed to more-secrets October 17, 2022 15:22 Inactive

PR comments

77d3402

benmoriceau temporarily deployed to more-secrets October 17, 2022 15:45 Inactive

Add cron micronaut environemnt

4e6ab3f

benmoriceau requested a review from jdpgrailsdev October 17, 2022 16:21

benmoriceau temporarily deployed to more-secrets October 17, 2022 16:23 Inactive

Merge branch 'master' into bmoric/remove-dep-server-worker

9f8c233

benmoriceau temporarily deployed to more-secrets October 17, 2022 17:29 Inactive

gosusnp approved these changes Oct 17, 2022

View reviewed changes

benmoriceau merged commit 7a71c55 into master Oct 17, 2022

benmoriceau deleted the bmoric/remove-dep-server-worker branch October 17, 2022 19:35

octavia-squidington-iii mentioned this pull request Oct 20, 2022

Bump Airbyte version from 0.40.15 to 0.40.16 #18268

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bmoric/remove dep server worker #17894

Bmoric/remove dep server worker #17894

benmoriceau commented Oct 12, 2022 •

edited

Loading

github-actions bot commented Oct 12, 2022

jdpgrailsdev Oct 17, 2022

benmoriceau Oct 17, 2022

jdpgrailsdev Oct 17, 2022

benmoriceau Oct 17, 2022

benmoriceau Oct 17, 2022

jdpgrailsdev commented Oct 17, 2022

benmoriceau commented Oct 17, 2022

Bmoric/remove dep server worker #17894

Bmoric/remove dep server worker #17894

Conversation

benmoriceau commented Oct 12, 2022 • edited Loading

What

Recommended reading order

github-actions bot commented Oct 12, 2022

jdpgrailsdev Oct 17, 2022

Choose a reason for hiding this comment

benmoriceau Oct 17, 2022

Choose a reason for hiding this comment

jdpgrailsdev Oct 17, 2022

Choose a reason for hiding this comment

benmoriceau Oct 17, 2022

Choose a reason for hiding this comment

benmoriceau Oct 17, 2022

Choose a reason for hiding this comment

jdpgrailsdev commented Oct 17, 2022

benmoriceau commented Oct 17, 2022

benmoriceau commented Oct 12, 2022 •

edited

Loading