Complete feature tests for RAP physics

[climbfuji]

Description

PR #937 adds several regression tests based on the RAP physics, including feature tests (restart, decomp, 2threads, ...). The following test fails with the RAP suite + GFS SFCDIFF:

rap_sfcdiff_decomp

The following tests fail with the RAP suite (default MYNN SFC):

rap_restart
rap_decomp

It seems like that the decomp issue is unrelated to the surface layer scheme, therefore fixing it for one of the tests will likely fix it for the other, too. Once the restart issue for the RAP (MYNN SFC) test is fixed, all RAP SFCDIFF feature tests can be removed (except for rap_sfcdiff_debug).

Also, when #937 was merged, the 2thread tests rap_2threads were turned off on jet.intel and wcoss_cray (because of wrong task geometry settings I think). They ran fine on all other platforms. It should be checked if they can be turned on with the correct settings (tasks per node etc.).

[arunchawla-NOAA]

there were a number of PRs merged related to this issue. @climbfuji is this still a problem ?

[climbfuji]

@arunchawla-NOAA Yes, there is still a problem with b4b reproducibility when changing the MPI decomposition.

[climbfuji]

@junwang-noaa @SamuelTrahanNOAA @grantfirl I started working on this. The decomp differences are definitely coming from the Grell Freitas scheme. When I return immediately from that scheme (after setting errmsg and errflg), differences are zero after two time steps. If I run GF, results are different after two time steps when changing the MPI decomposition. Stay tuned.

[climbfuji]

The remaining RAP (GF) decomp b4b issue is fixed in #1243 and associated PRs for fv3atm and ccpp-physics :-)

[SamuelTrahanNOAA]

This should be combined with #1257 which enables the full feature set for hrrr_control. Fortunately, that one needs no code changes.

[SamuelTrahanNOAA]

I've merged Dom's fix for this issue with my PR here:

#1257

[SamuelTrahanNOAA]

@climbfuji Your bugfix has been merged. Can you confirm the issue is resolved, and close it if it is?