Support Query Ramp-up

[gingerwizard]

Currently, after a warmup period, all query tasks are executed together at the same time. After the first execution, they respect their configured scheduler for subsequent executions.

In cases with a high number of tasks (e.g. executing using a parallel block) this absence of a ramp up, can produce an initial extremely high load and result in excessive utilization. This is best illustrated through an example:

Even considering the warmup period (darker in the first visual) it's apparent we suffer from a spike before the model stabilizes.

We should consider both short term improvements here e.g. maybe tasks call their scheduler before executing the first time?,
and longer-term possible improvements e.g. a ramp-up period.

@danielmitterdorfer

[danielmitterdorfer]

We should consider both short term improvements here e.g. maybe tasks call their scheduler before executing the first time

I had a look at this proposal and the change for this is actually pretty small:

rally/esrally/driver/driver.py

Lines 1742 to 1771 in 3875968

	async def __call__(self):
	next_scheduled = 0
	if self.task_progress_control.infinite:
	param_source_knows_progress = hasattr(self.params, "percent_completed")
	self.task_progress_control.start()
	while True:
	try:
	# does not contribute at all to completion. Hence, we cannot define completion.
	percent_completed = self.params.percent_completed if param_source_knows_progress else None
	#current_params = await self.loop.run_in_executor(self.io_pool_exc, self.params.params)
	yield (next_scheduled, self.task_progress_control.sample_type, percent_completed, self.runner,
	self.params.params())
	next_scheduled = self.sched.next(next_scheduled)
	self.task_progress_control.next()
	except StopIteration:
	return
	else:
	self.task_progress_control.start()
	while not self.task_progress_control.completed:
	try:
	#current_params = await self.loop.run_in_executor(self.io_pool_exc, self.params.params)
	yield (next_scheduled,
	self.task_progress_control.sample_type,
	self.task_progress_control.percent_completed,
	self.runner,
	self.params.params())
	next_scheduled = self.sched.next(next_scheduled)
	self.task_progress_control.next()
	except StopIteration:
	return

We basically need to call the scheduler's next method before the generator yields insted of afterwards.

[gingerwizard]

This maybe has the potential to cause a deviation in nightly performance @danielmitterdorfer if the number of requests isn't high - especially around our 95th and 99th percentile. If anything it might bring greater stability to the results 🤞

[danielmitterdorfer]

I don't think that it would cause a deviation in the nightlies for most tracks because we only test with a single client and the effect only shows up with multiple clients. But for more complex benchmarks this is more likely an issue.