Continue Sessions using API from esmvalcore.experimental

[lukruh]

Is it possible to continue previous recipe runs? i.e. create a session from an output folder?
Running esmvaltool from the command line allows to continue previous sessions by rerunning specific tasks from the output folder. I'm looking for something similar to be called from within a jupyter notebook. Using CFG.start_session() a new folder is created and all ancestors need to be run again.

[bouweandela]

The options from the command line can also be specified in config-user.yml and the esmvalcore.config.Sesssion you're using, so you should be able to do something like

CFG.start_session()
session['resume_from'] = ["/path/to/previous/run", "/path/to/other/previous/run"]

or possibly

from pathlib import Path
CFG.start_session()
session['resume_from'] = [Path("/path/to/previous/run"), Path("/path/to/other/previous/run")]

(I did not try this myself). Does that solve your problem?

[lukruh]

The resume_from option is very useful. I didn't realized that the command line options can be passed to the Session yet. Thanks for the example.

[dhohn]

Hi @lukruh , @bouweandela ,

I have followup questions as I would like to do that same thing as OP. @lukruh did you end up using the resume_from successfully?

I have been trying to use it today and ran into some trouble:
How do I load the session correctly?

1. When using the experimental API (esmvalcore.experimental.recipe.Recipe) I found I have to use the "private" _load method.
2. When using the new (?) "private" API (esmvalcore._recipe.recipe) I can use the read_recipe_file function.

Which is recommended?

Also in both cases I first have to manually make sure that the new session_dir and run_dir exists i.e. like this:

session.session_dir.mkdir(parents=True)
session.run_dir.mkdir()

Im looking forward to when notebook support gets promoted to a first class feature. Right now theres a few little issues like this one that make life a bit inconvenient.

[bouweandela]

How do I load the session correctly?

If you want to configure the session yourself, you can do so as described here and then you can provide it as an argument to the esmvalcore.experimental.Recipe.run method. Using functions/modules that start with an _ is not recommended because these should be considered internal and can be changed/removed without notice from one version of the tool to the next.