cloud environment may restart workflows on deploy

[jrhizor]

Despite our tests for exactly this case in OSS we are seeing workflows restart.

https://airbytehq-team.slack.com/archives/C02TXQ020QM/p1645216105079419 for an example.

I need to dig in and see exactly what is failing here and why the tests are not representative. Maybe the Helm deploy mechanism terminates pods differently than the scale down in the tests.

[jrhizor]

@cgardens fyi

[cgardens]

@jrhizor thanks for the heads up. agreed that this is a high prio thing that needs to be figured out by the launch.

[jrhizor]

This morning I figured out why each of the test cases was having the behavior observed:

Test Case	Passes (prod)	Passes (new)	Why?
KILL_ONLY_NON_SYNC	Yes	Yes	This is the easiest to handle, and what we were testing previously. If the sync workflow isn't terminated the connection manager resumes as expected.
KILL_BOTH_SAME_TIME	No	Yes	Here the manager is still killed slightly before the other, which means that depending on the sync retry policy it can "reconnect" to the sync workflow.
KILL_SYNC_FIRST	No	Yes	Here the sync is killed, but the manager doesn't detect it with the new configuration because it retries within the timeouts.
KILL_NON_SYNC_FIRST	No	Yes	The same thing happens in the opposite direction, it retries but within timeouts.
KILL_ONLY_SYNC	No	No	When you kill the sync only, there is no timeout-based grace period. Currently working on how to fix.

[jrhizor]

Updates are on #10565. It's pretty well tested at this point. Will do the release in a few hours and close this.

[jrhizor]

Verified with https://cloud.airbyte.io/workspaces/6a9fca7e-8af0-40b6-95ea-b23efb3245b0/connections/00dfafc1-83d1-4ae1-b4cc-a4615e25cacd/status (a failure but the sync continued properly throughout).