Add mesh based cap for use by S2SW and HAFSW

[DeniseWorthen]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

Implement a new mesh-based NUOPC cap for the S2SW and HAFSW applications which is restart reproducible and which can be run in debug mode. The new mesh-based cap allows the WW3 component the ability to run as shel or multigrid via configuration namelist. Currently only a single grid is supported when run as multigrid.

The default_vars have been updated as follows:

• for the coupled S2SWA app:

export FIELD_TABLE=field_table_thompson_noaero_tke_GOCART
export NEMS_CONFIGURE=nems.configure.cpld.IN
export MULTIGRID=false
export CPLCHM=.true.

• for the HAFSW app:

export MESH_WAV=mesh.hafs.nc
export MULTIGRID=false

The new WW3 cap utilizes a ww3_shel.inp.IN file in the parm directory. Defaults for both S2SWA and HAFSW are:

export WW3RSTDTHR=3
export WW3OUTDTHR=3
export DT_2_RST="$(printf "%02d" $(( ${WW3RSTDTHR}*3600 )))"
export DTFLD="$(printf "%02d" $(( ${WW3OUTDTHR}*3600 )))"
export DTPNT="$(printf "%02d" $(( ${WW3OUTDTHR}*3600 )))"
export WW3GRIDLINE="'ww3'  'no' 'CPL:native' 'CPL:native' 'CPL:native' 'no' 'no' 'no' 'no' 'no'   1  1  0.00 1.00  F"
export UNIPOINTS='points'

• for S2SWA, the following are also used in the ww3_shel.inp.IN file

export INPUT_CURFLD='T F     Currents'
export INPUT_ICEFLD='T F     Ice concentrations'

• for HAFSW, the following are also used

export INPUT_CURFLD='F F     Currents'
export INPUT_ICEFLD='F F     Ice concentrations'

Additional changes to tests include

• The cpld c384 tests (control, restart) are dropped and replaced by a the a new restart test for the cpld_bmark_p8 test.

• The cpld_bmark_p7 and cpld_control_p7_rrtmgp are dropped since they rely on WW3 using connectors for the global model.

• For each platform, the following tasking sets are removed: cpl_c96, cpl_c384,cpl_bmrk_aero, bmrk_mpi. The cpl_c192 has waves tasks added in each case.

• Only two nems.configure options for the global coupled model are retained; the noaero version is used only by the GNU cpld test. See issue Segfaults when running cpld aerosol model with GNU 9 and 10 #1115

Issue(s) addressed

• fixes Update WW3 to use new mesh based cap for S2SW and HAFSW #1102
• fixes add waves into debug tests #229
• fixes Restart reproducibility for S2SW (was Target improvements/changes for Prototype 5) #739
• fixes hafs_regional_atm_ocn_wav seems to have incorrect global wind fields at startup #993
• fixes DT_INNER is set incorrectly in cpld_control_c192_p8/cpld_restart_192_p8 tests #1134
• fixes CMake build for WW3 fails when compiling S2SW with -DDEBUG=ON #1133

Testing

All baselines for app=NG-GODAS using the 1deg MOM6 grid will change because of the closure at j=1.

All baselines for app=S2SWA and app=HAFSW will change. For the S2SWA app, this is because of the switch to the CMEPS mediator as well as the grid change (eg from a regular 30m grid to the MOM6 tripole grid).

For the HAFSW app, the baseline will change because of the switch from the existing WW3 cap based on a grid to the mesh based cap. The mesh file provided to the HAFSW application has slightly different (O~10-6) lat/lon values than the grid produced in the existing cap. For example, the latitude difference between the existing cap and the mesh cap through the middle of the domain is shown here:

Fields exported by Wave model are shown in the following google slides

See issue #739 and issue #993 for additional figures.

This PR branch was tested using a staged input-data directory on hera at cb1d019. All tests passed except for the expected failures (all cpld, all datm tests at 1deg, hafs tests with waves). Tests were repeated after the RRTMGP and HAFS moving nest PRs were merged and confirmed that only the expected tests failed.

Updated testing

Testing at 0cf22ee after updating (but not pushing) the WW3 mesh cap branch with the changes in PR #1228 gave the following results:

cheyenne.intel : All jobs run to completion. Baselines fail only for the expected cases (cpld and hafs tests using waves). Tests no longer fail for the DATM cases, since the j=1 fix has been merged (PR #909).

cheyenne.gnu: All jobs run to completion. Baselines fail only for the cpld_control_noaero_p8 and cpld_debug_noaero_p8 tests, which now contain the wave component.

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3
• wcoss2
• opnReqTest for newly added/changed feature. Tested at 9e1fec2. The following tests passed: cpld_control_p8 thr,dcp,rst,fhz,dbg and cpld_bmark_p8 thr,dcp,rst.
• CI

Dependencies

Requires:

• WW3 PR (Add mesh cap  NOAA-EMC/WW3#641)
• PR Close MOM6 1deg at j=1; Update input-data for P8c #909
• PR P8c ICs #1108
• Fix for Issue S2SW noaero test fails in Debug mode for GNU #1139
• For S2SW with GNU, requires fix for unformatted file open statements in WW3. See issue Segfaults when running cpld aerosol model with GNU 9 and 10 #1115
• Reduction of bmark_p8 test length and/or walltime on Cheyenne.intel
• Fix for writing GoCart history files at specified interval, currently only obtaining them at 06h intervals
• WW3 mod_def files with obstruction grids and/or timestep modification for MOM6 tripole grids. A new WW3 input directory WW3_input_data_20220418 has been added to all platforms in input-data-20220414.
• Requires resource testing on all platforms to ensure the tests don't exceed ~20 mins wall clock time

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
[BL] Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/884243400/20220530100019/ufs-weather-model
[BL] Error: Test cpld_control_c192_p8 002 failed in run_test failed
Please make changes and add the following label back: jet-intel-BL

[MinsukJi-NOAA]

@DeniseWorthen Can you please make this simple change for CI? This does not affect any RT results.

diff --git a/tests/ci/ci.test b/tests/ci/ci.test
index 2fea94f8..9feda870 100644
--- a/tests/ci/ci.test
+++ b/tests/ci/ci.test
@@ -2,5 +2,5 @@ control
 thr mpi dcp rst bit dbg
 regional_control
 thr dcp
-cpld_control_c96_noaero_p8
+cpld_control_noaero_p8
 rst dbg

[junwang-noaa]

Is this test for ww3 multigrid?

[DeniseWorthen]

Yes, this runs the wave model in multi (but with a single grid). It should reproduce the shel run. But there is an issue which we found after Jessica created the new mod_def files with the flag flck=true (which it should be). We found that the shel and multi cases diverge at the third timestep (#1131 (comment)) so until the issue can be resolved, I've turned off this test.

[junwang-noaa]

@JessicaMeixner-NOAA @BinLiu-NOAA @MinsukJi-NOAA All the RT passed, would you please review the PR? Thanks

[JessicaMeixner-NOAA]

@DeniseWorthen are you ready for the WW3 PR to be merged?

[DeniseWorthen]

@JessicaMeixner-NOAA I believe so. It should be up-to-date with the dev/ufs-weather-model branch. Let me know if you see any issues.

[JessicaMeixner-NOAA]

@DeniseWorthen the WW3 PR has been merged.

[BrianCurtis-NOAA]

Hash's look good.