Job Coordination Improvement Proposal

[pyalex]

Is your feature request related to a problem? Please describe.

There's some technical debt in ingestion part mostly related to Job Coordination: job is being restarted on schema update, this restart can take arbitrary amount of time, hence we had to split jobs as much as possible to minimize mutual disruption (on restart). This negatively affects our UX and also restrict us from using Dataflow resources more efficiently, which currently leads to very high spendings on Dataflow.

Describe the solution you'd like

I drafter proposal that addresses those issues and suggest new design in communications between core service and ingestion pipeline
https://docs.google.com/document/d/1gqkCWZUyVBIU8OKhxIhIf1BBd3JrbOx2WnXrpGLFCAc/edit#heading=h.wigpvke4im4g

[woop]

Might be relevant to the following folks. @ches @Yanson @dr3s @idahoakl

TL;DR: We want to reduce the responsibility of the Job Coordinator to spin up new jobs when new feature sets are registered (or changed). The existing jobs should accommodate changes to the schemas.

Please flag any concerns with this approach if you have any.