-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: refresh actor configuration and state between sync attempts #21629
Conversation
Airbyte Code Coverage
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay, I missed this PR. It LGTM, it doesn't seems that there is a concurrency issue with this PR. About the comment where you tagged me, I believe that it is some legacy code that should be removed. This shouldn't be part of your PR and I'll create an issue for for that.
…nc attempts"" (airbytehq#22281) * Revert "Revert "fix: refresh actor configuration and state between sync attempts (airbytehq#21629)" (airbytehq#22211)" This reverts commit 7978862. * fmt
What
This is part 1 of #20912, focused on removing the actor configuration and state from
JobSyncConfig
and moving into a newAttemptSyncConfig
at the Attempt level. It will also mean that config/state is fetched when building the sync input rather than retrieving it from the job's JobSyncConig.Two issues are fixed with these changes:
However, it does not address when the CHECK that can run before the SyncWorkflow makes changes to the configuration. This will be addressed via a follow up PR (part 2) via the solutions outlined in the tech spec for #20912, specifically the part on changing the ConnectionManagerWorkflow activities that generate inputs.
How
AttemptSyncConfig
pojo withsourceConfiguration
destinationConfiguration
andstate
fields.JobSyncConfig
attempt_sync_config
column to theattempts
table.AttemptSyncConfig
when generating the sync inputs by fetching the latest actor configuration + state, and write the values to the above column.