Improve scheduling by interleaving setup and teardown of an experiment

[bodokaiser]

ARTIQ Feature Request

Problem this request addresses

Scheduling of experiment runs is not efficient, i.e., we "waste" about 1 second between experiment runs.

Describe the solution you'd like

Currently, the scheduler worker runs an experiment run's setup and teardown steps, e.g., exp.prepare and write_results, sequentially.

Can we have a worker thread for each stage? Could we also add a precompilation step?

[sbourdeauducq]

Currently, the scheduler worker runs an experiment run's setup and teardown steps, e.g., exp.prepare and write_results, sequentially.

What do you expect?

The experiment needs to be prepared before it can run.
Results need to have been obtained before they can be written.

If you have two different experiments then those steps should already run in parallel.

Precompilation is already supported, please read the manual.

[architeuthidae]

Currently, the scheduler worker runs an experiment run's setup and teardown steps, e.g., exp.prepare and write_results, sequentially.

It could hardly be the other way around. On the other hand, experiment stages are already pipelined (and there can be multiple pipelines at any one time.) See the manual. There is also already support for precompilation.

[architeuthidae]

Oh, race condition. Excuse me.

[sbourdeauducq]

i.e., we "waste" about 1 second between experiment runs.

What operating system is this?
Each experiment runs in a different worker process. How much of that second is just creating a process?
Though even then, it does not seem worthwhile to optimize since this only affects the first experiment in a sequence, then the rest gets pipelined.

[sbourdeauducq]

If it really matters (but you should make the case for it, everything is a trade-off against code complexity) it should be possible to create some worker processes in advance.

[bodokaiser]

What do you expect?

The experiment needs to be prepared before it can run. Results need to have been obtained before they can be written.

Yes, but I can prepare for the next experiment while writing the results for the previous one.

If you have two different experiments then those steps should already run in parallel.

Why not extend it for the same experiments (with different argument overwrites)?

Precompilation is already supported, please read the manual.

Yes, but how would I tell the scheduler to exploit precompile? Is it as simple as

class Experiment(EnvExperiment):

  def prepare(self):
    self._run = self.precompile(self.run)

  def run(self):
    self._run()

?

i.e., we "waste" about 1 second between experiment runs.

What operating system is this? Each experiment runs in a different worker process. How much of that second is just creating a process? Though even then, it does not seem worthwhile to optimize since this only affects the first experiment in a sequence, then the rest gets pipelined.

We are using Linux amd64. Yes, you are right, a starting point is get a better understanding of the actual bottlenecks. I can provide an analysis in a few weeks (also testing the new NAC3 compiler).

[dnadlinger]

Yes, but I can prepare for the next experiment while writing the results for the previous one.

This should already be the case. The next-to-be-run experiment is already prepared while the currently active one is still running.