Optimise concurrency for scale

[zaneb]

Currently the default value for the $BMO_CONCURRENCY variable is 3. We chose this value because (a) it wasn't 1 (so in testing we would not end up relying on only 1 BMH being reconciled at a time), and (b) it seemed like as good a number as any. With the benefit of more data, we can say a bit more about the optimal value.

We currently limit the number of nodes that can be provisioned at one time ($PROVISIONING_LIMIT) to 20 to limit the load on Ironic. Note that enforcement of the provisioning limit cannot be made thread-safe, so while it isn't a hard limit, we should endeavour to keep $BMO_CONCURRENCY << $PROVISIONING_LIMIT to avoid smashing right through it. Most of the ironic provisioning work happens asynchronously from the perspective of the baremetal-operator; we only schedule a reconcile to check on it every 10s.

However, before a new Host can start provisioning, it must pass through a number of states, including those that don't do ~anything (e.g. match profile, or inspecting when inspection is disabled). If a large number of Hosts are added simultaneously, they will have a tendency to go through this process in lockstep, since as each is advanced it will go to the back of the queue before it will get processed again. Ideally we want at least 20 Hosts to be available for provisioning as quickly as possible.

I wrote a little simulation to experiment with different settings. If we assume 1000 new Hosts (why not) added at once and 20 reconcile steps (wild guess) each taking around 100ms (pretty accurate when there is no work to do), then it would take more than 10 minutes for the first 100 nodes to become available for provisioning. Real-world testing with 1000 nodes created simultaneously shows that indeed, very few hosts are provisioned in the first 10-20 minutes. This can be improved by increasing the number of concurrent reconciles allowed.

Another option is to add some random small delay when requeuing, instead of requeuing without delay. As each reconcile step is worked through, the Hosts should tend to spread out in time, so that some reach the provisioning state sooner than others. Using the same simulation, the optimal amount of jitter to add is:

1.9 * <Total number of hosts> * <Average reconcile time> / <Number of concurrent reconciles>

(I haven't determined the mathematical significance of the value 1.9, but it's accurate to 2s.f.)

Plotting the effect of both interventions:

Adding jitter reduces the time for the first 100 hosts to go through 20 reconcile steps by about 10-15%, but increasing the concurrency has a much, much larger effect. A default value of 8 for $BMO_CONCURRENCY would seem to capture most of the returns at this scale.

[furkatgofurov7]

/kind feature