Use Floe wait for event driven updates to workflows

[agrare]

Currently when running a WorkflowInstance we will continue to step through a workflow until it would block, once it would block we queue another check for a time in the future (currently it is 10 seconds from Time.now). This is a compromise in the sense that quick running containers will likely finish well under that and be waiting for a while, and long running containers will be checked multiple times before they complete.

Floe has a .wait interface (both on the Floe::Workflow and the Floe::Workflow::Runner which will block until there is an update that can be acted upon.

Requirements:

• Wait for events from the underlying docker runner (podman, kubernetes, or docker)
• Wait for time-based events (e.g. a Wait state or a Task Retrier)
• Run a periodic check in case we missed or mishandled an event to ensure workflows don't hang indefinitely
• Handle workflows being added and removed while running (I don't think we can use Floe::Workflow.wait([workflows, ...]) for this reason)
• Handle workflows already running at startup (e.g. we crashed and restarted). For this we need to know which workflows "our" server is responsible for running (add miq_server_id to configuration_script?)

[kbrock]

Is "Handle workflows being added and removed while running" the same thing as "Handle new workflow requests" (via MiqQueue)?

One design aspect, don't drop workflows on the ground if this process crashes.
Not sure if we need to handle Map/Parallel concepts at this time

[agrare]

Is "Handle workflows being added and removed while running" the same thing as "Handle new workflow requests" (via MiqQueue)?

Yes I believe so

One design aspect, don't drop workflows on the ground if this process crashes.

That is what I was trying to capture with "Handle workflows already running at startup (e.g. we crashed and restarted)."