Wrapper logic: fre pp run

[cwhitlock-NOAA]

The thing-currently-named-wrapper (wrapper.py) works for testing fre-cli, when we have all history files already available at the time of running. However, when we get to running production jobs, the models will be sending over bundles of history files and post-processing them in parallel. This breaks some of the logic currently in the wrapper flow - in particular, the assumptions that there's not already a pre-existing experiment belonging to the same user that the current set of history files is being added to, and the assumption that there's not already an experiment with that name running.

The logic we need is encapsulated in a flowchart at the end of this issue; this breaks it down by tool.

The tool:
Fre pp run has the most complicated logic update of the 4 tools touched by the wrapper production logic. It needs to check for prior installed experiments (not just checked out) and perform different cylc commands depending on the status of the installed experiments.

fre pp run:

[ ] Is the -e -p -t workflow installed? If not, install. (current behavior)

• test: the current behavior. If not installed, install; probably uses some of the same test logic that we used for the fre pp checkout tests
  [ ] If it is installed, does the workflow match the existing config? Check the gitlab fre2/workflows codebase too.
• test: not that much more than a diff plus the commands from fre checkout to see if the commit history matches
  If answer is yes:
  [ ] If the workflow is not running, start running - cylc run
• test: see note
  [ ] If the workflow is already running (i.e. a previous set of history files started the experiment) - cylc trigger
• test: see note
  If answer is no:
  [ ] Do no automatically exit - prompt user for more info. The current experiment may be overriding a previous configuration.
• test: see if pytest has a function equivalent to piping "yes" for a command-line prompt
  [ ] If we want new config and nothing is running - cylc run
• test: see note
  [ ] If we want new config and there's already an experiment running (i.e. you noticed a config error too late) - cylc reload
• test: see note

Note: I'm not sure how to do the tests for this section - we need a lot of experiments running, with very specific behavior that is tied to how long an experiment has been running (i.e. whether it has completed) or what the user has been doing (overriding a previous configuration). I'd like to divide this up into smaller chunks, but I'm not sure how.

[ceblanton]

part of 2024.01

[ceblanton]

the green part of the flow chart only for 2024.01