Regression tests

[JessicaMeixner-NOAA]

When CMAKE is added to WW3 there will be a possibility to compile multiple builds at the same time. With this new capability, could bring the possibility of updating/improving the way we run the matrix.base full set of regression tests. We can just make small changes to incorporate CMAKE, but I thought this might could be an opportunity to step-back and ask a larger question of how the regression tests are run in general, beyond just the build an am curious if others have any ideas.

[aliabdolali]

We can get rid of matrix_divider and use the parallel build and run in one single job, but it requires some workflow manager, available across platforms, right?

[JessicaMeixner-NOAA]

or the matrix divider uses different build directories instead of creating the different model directories.

I'd be hesitant to use a workflow manager when we'd like to just want to minimize this to a few jobs, but perhaps we have one job that runs first that builds all the exes, then we run the jobs? Or something else? My thought it we don't want to lose the capability of the "run one example start to finish" but also maybe there are better ways in terms of "one step builds", the next step runs many serial jobs at the same time for all the prep steps, then you run multi/shel, then you do post processing. My thoughts are: faster so that the full set finishes quicker and two use resources more efficiently.

[JessicaMeixner-NOAA]

@mickaelaccensi suggested an idea to have build directories for each distinct set of switches, and then just point to those build directories. His idea was to use the switch names in the directory name but I think that would be too long of a directory name for some platforms so we might have to figure out a naming convention that would shorten the directory name. It might also be worthwhile to see where we could make updates/edits to our existing switch lists to reduce the number of differences when/where appropriate.