Replace Scheduler with a Workflow Engine

[cgardens]

Tell us about the problem you're trying to solve

• We are rewriting features that exist in any good workflow engine. This is bad because it is slowing us down or making us make trade offs on features in the scheduler that we shouldn't have to make.
• Existing issue: We want to be able to gracefully cancel a running job. This opens up the ability to greatly simplify the flow for upgrading airbyte versions. But we're gated on writing this feature which would exist in most workflow engines.
• Existing issue: If airflow is kill -9'ed while a job is running it is unclear that it will recover gracefully when it is started back up. That's because our state machine doesn't have anyway of resolving a job that is in state RUNNING but isn't actually running. We can add code to handle this case, but again we are re writing functionality that is done in any workflow engine.

Describe the solution you’d like

• I think airflow is the default choice here, but I'm not picky if there's a good case for some other workflow engine (e.g. cadence).
• tbd list full feature set we are looking for.

[cgardens]

Goal for this week is a tech spec.

[jrhizor]

Planning on working on this over the weekend. Want to finish this out.

[jrhizor]

Depends on:

• Separate migration code from JobPersistence #2169
• [EPIC] Support cron strings as the primary scheduling method in the backend #2170
• Change BE/FE interface for syncs to only have an async call #2171
• Write StandardSyncSummary and State from sync jobs into a persistence #2172
• Write StandardSyncSummary and State from sync jobs into a persistence #2172
• Attempts should run on the same input #2173
• improving logging interface for workers #2175
• use exceptions for job error reporting and outputs for the actual job output #2176
• improve worker construction interface #2177
• Move scheduling (i.e. getNextJob) logic outside of JobPersistence #2178
• Refactor JobSubmitter/JobCreator/JobPersistence interface #2179

These are all available under the temporal label temporal

After that the changes #2130 should be easy to slot in as an atomic change:

• switching the new interface to Activities/Workflows
• rename the scheduler container to worker
• testing only that functionality
• tune settings
• release temporal
• add cancellation scope
• release cancellation feature