Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bmoric/temporal cleaning cron #16414

Merged
merged 142 commits into from
Sep 26, 2022
Merged

Bmoric/temporal cleaning cron #16414

merged 142 commits into from
Sep 26, 2022

Conversation

benmoriceau
Copy link
Contributor

@benmoriceau benmoriceau commented Sep 7, 2022

What

Update the cron project in order to properly inject the temporal client in the scheduled service.

jdpgrailsdev and others added 30 commits September 7, 2022 13:51
* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue
* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

* tmp

* tmp

* View diff

* Move part of the temporal client

* tmp

* copy Temporal Utils test

* Uniq Temporal Utils

* Uniq Temporal Workflow Utils

* Move CancellationHandler

* Move commons-temporal to being shared between platform, connector and CLI

* Rm worker dependency from cron

* Fix build

* Update with right value for cron.

* Fix dep conflict
@benmoriceau benmoriceau temporarily deployed to more-secrets September 26, 2022 21:01 Inactive
@benmoriceau benmoriceau temporarily deployed to more-secrets September 26, 2022 21:44 Inactive
@benmoriceau benmoriceau merged commit bbb59d8 into master Sep 26, 2022
@benmoriceau benmoriceau deleted the bmoric/temporal-cleaning-cron branch September 26, 2022 22:14
benmoriceau added a commit that referenced this pull request Sep 26, 2022
benmoriceau added a commit that referenced this pull request Sep 26, 2022
benmoriceau added a commit that referenced this pull request Sep 27, 2022
benmoriceau added a commit that referenced this pull request Sep 27, 2022
* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"

This reverts commit 7516679.

* Generate an artifact

* Change missing property from value to property
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Add the injected temporal client to airbyte-cron

* Update cron to make it work

* Remove useless Bean factory

* PR comments

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* Env vars for log/state storage type

* Remove use of Optional in bean declarations

* Fix typo in config property name

* Support Temporal Cloud namespace

* Change to @value

* Use correct value for conditional check

* Upgrade Micronaut

* Fix merge conflict

* Formatting

* Add missing env var

* Use sync task queue environment variable

* Handle sync task queue as set

* format and force http

* Handle case where sync task queue is empty

* Add correct path to config property

* Remove unused import

* Remove conflict

* Remove unused parameter

* Formatting

* Use pattern for condition process factory beans

* Cleanup

* PR feedback

* Revert hack for testing

* Fix temporal restart by status (airbytehq#16447)

* Update application.yml

* Re add worker dep

* Add missing env var

* PR comments

* Bmoric/move temporal client (airbytehq#16778)

* tmp

* tmp

* View diff

* Move part of the temporal client

* tmp

* copy Temporal Utils test

* Uniq Temporal Utils

* Uniq Temporal Workflow Utils

* Move CancellationHandler

* Move commons-temporal to being shared between platform, connector and CLI

* Rm worker dependency from cron

* Fix build

* Update with right value for cron.

* Fix dep conflict

Co-authored-by: jdpgrailsdev <[email protected]>
Co-authored-by: pmossman <[email protected]>
Co-authored-by: Jonathan Pearlin <[email protected]>
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
* Revert "Revert "Bmoric/temporal cleaning cron (airbytehq#16414)" (airbytehq#17192)"

This reverts commit 7516679.

* Generate an artifact

* Change missing property from value to property
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Add the injected temporal client to airbyte-cron

* Update cron to make it work

* Remove useless Bean factory

* PR comments

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (airbytehq#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* Env vars for log/state storage type

* Remove use of Optional in bean declarations

* Fix typo in config property name

* Support Temporal Cloud namespace

* Change to @value

* Use correct value for conditional check

* Upgrade Micronaut

* Fix merge conflict

* Formatting

* Add missing env var

* Use sync task queue environment variable

* Handle sync task queue as set

* format and force http

* Handle case where sync task queue is empty

* Add correct path to config property

* Remove unused import

* Remove conflict

* Remove unused parameter

* Formatting

* Use pattern for condition process factory beans

* Cleanup

* PR feedback

* Revert hack for testing

* Fix temporal restart by status (airbytehq#16447)

* Update application.yml

* Re add worker dep

* Add missing env var

* PR comments

* Bmoric/move temporal client (airbytehq#16778)

* tmp

* tmp

* View diff

* Move part of the temporal client

* tmp

* copy Temporal Utils test

* Uniq Temporal Utils

* Uniq Temporal Workflow Utils

* Move CancellationHandler

* Move commons-temporal to being shared between platform, connector and CLI

* Rm worker dependency from cron

* Fix build

* Update with right value for cron.

* Fix dep conflict

Co-authored-by: jdpgrailsdev <[email protected]>
Co-authored-by: pmossman <[email protected]>
Co-authored-by: Jonathan Pearlin <[email protected]>
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
* Revert "Revert "Bmoric/temporal cleaning cron (airbytehq#16414)" (airbytehq#17192)"

This reverts commit 7516679.

* Generate an artifact

* Change missing property from value to property
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform area/server area/worker Related to worker
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants