[develop] Add three UFS Case Studies to WE2E testing process

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

Adding three additional UFS Case Studies (2019 Hurricane Barry, 2019 Halloween Storm, and 2020 July CAPE) to the workflow end-to-end testing process as required by one of our SI features. The UFS Case studies test yamls were created and added to the gaea-c5, derecho, and hera.gnu.com coverage suite files as well as the comprehensive suite files. Removed Cheyenne related logic from current WE2E UFS Case Study (2020_CAD) and extended fcst wall time for it.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
• orion.intel
• hercules.intel
• cheyenne.intel
• cheyenne.gnu
• derecho.intel
• gaea.intel
• gaeac5.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform) ran test on PW AWS
• Jenkins
• fundamental test suite ran on PW AWS
• comprehensive tests (specify which if a subset was used)

2020 CAPE Results
CAPE results from the FV3_GFS_v16 physics suite compare well with the GFS_v15p2 physic suite used in the case study.

2019 Halloween Results
10m surface winds match well with the RAP Analysis used in the case study.

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

[RatkoVasic-NOAA]

new tests passed on Hera:

All 3 experiments finished
Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAPE_20240226235238                                           COMPLETE              33.44
2019_hurricane_barry_20240226235240                                COMPLETE              32.83
2019_halloween_storm_20240226235242                                COMPLETE              32.32
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              98.59

Detailed summary written to /scratch2/NCEPDEV/fv3-cam/Ratko.Vasic/1043/expt_dirs/WE2E_summary_20240227021304.txt

[RatkoVasic-NOAA]

Tests passed. Approved.

[MichaelLueken]

@EdwardSnyder-NOAA - These changes look good to me! I was able to successfully run the new tests on Hercules:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
2019_halloween_storm_20240227085827                                COMPLETE              31.87
2019_hurricane_barry_20240227085828                                COMPLETE              32.72
2020_CAPE_20240227085829                                           COMPLETE              32.33
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              96.92

Approving now.

[MichaelLueken]

The Jenkins tests on Hera GNU, Hera Intel, Hercules, and Orion have successfully passed.

Two WE2E coverage tests on Jet failed - custom_ESGgrid_Great_Lakes_snow_8km and get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h. These failures are due to get_obs_nohrsc and make_lbcs jobs hanging, respectively. Reruns of these two tests are happening now.

There was a disconnect failure on Derecho, but the tests were resubmitted and all successfully passed.

Once the Jet tests pass, I will move forward with merging this PR.

[MichaelLueken]

The summary for Derecho WE2E coverage test:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240228061703                      COMPLETE              25.95
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              43.15
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022806170  COMPLETE              50.00
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240228  COMPLETE              35.16
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              20.82
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022806171  COMPLETE              45.77
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              25.66
pregen_grid_orog_sfc_climo_20240228061718                          COMPLETE              17.97
specify_template_filenames_20240228061720                          COMPLETE              20.64
2019_hurricane_barry_20240228061722                                COMPLETE              39.50
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             324.62

The final summary for Jet WE2E coverage tests:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community_20240227181256                                           COMPLETE              20.36
custom_ESGgrid_20240227181323                                      COMPLETE              19.84
custom_ESGgrid_Great_Lakes_snow_8km_20240227181325                 COMPLETE              14.94
custom_GFDLgrid_20240227181330                                     COMPLETE              10.03
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202402  COMPLETE              14.00
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              62.53
get_from_HPSS_ics_RAP_lbcs_RAP_20240227181357                      COMPLETE              17.86
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240227181403  COMPLETE             237.64
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              40.14
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               7.93
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             509.09
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024  COMPLETE              10.64
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             965.00

Moving forward with merging this PR now.