Decomp test fails for coupled model in P7 configuration #813

DeniseWorthen · 2021-09-17T13:51:09Z

DeniseWorthen
Sep 17, 2021
Maintainer

The cpld_control_p7 test in the PR #765 branch does not pass the decomposition test. This test changes the layout for FV3 and compares against the control run.

The failure of the decomp test for the standalone with UGWPV1 turned on was identified in Issue #742. A PR #798 has been made which resolves the decomp failure with UGWPV1 in the standalone.

Testing has shown that the fix does not resolve the issue for the cpld_control_P7 test.

A sandbox setup (similar to that in Discussion #797) has been set up to debug the issue with the following changes:

The input namelists and configuration files for the base and decomp run differ only in input.nml. In the decomp case, the layout in input.nml has been changed to layout = 6,4.
CA has been turned off in both tests.
The run length has been shortened to 12 hours.

The FV3 branch from PR #798 has been used to compile the FV3 executable for the coupled model. The sandbox is located at /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/DECOMP96 . The initial results show that the following fields exported by the ATM do not reproduce with a decomp change:

63: RMS atmImp_Faxa_rain                 1.4918E-09            NORMALIZED  3.1154E-05
135: RMS atmImp_Sa_pbot                   1.0531E-05            NORMALIZED  1.0692E-10
151: RMS atmImp_Sa_shum                   4.7001E-11            NORMALIZED  5.1430E-09
160: RMS atmImp_Sa_tbot                   2.8200E-07            NORMALIZED  9.8108E-10
183: RMS atmImp_Sa_z                      1.0163E-08            NORMALIZED  9.6993E-10
465: RMS ocnExp_Faxa_rain                 5.7909E-10            NORMALIZED  1.8350E-05

These differences appear randomly on multiple tiles. There is no clear association with either land, ocean or fractional grid (coastal) points. The following shows the differences in the exported temperature_height_lowest at the end of the first fast coupling step:

The following have been tested but have not resolved the decomp failure:

zero-ing the tendencies by setting

<   do_ugwp_v1   = .false.
<   do_gsl_drag_ls_bl    = .false.
<   do_gsl_drag_ss       = .false.
<   do_gsl_drag_tofd     = .false.

compiling in debug mode
turning off fractional grid

A new standalone control_p7 has been added to this branch. The changes between this testing branch and the PR #765 branch are shown here

The FV3 in this branch needs to be switched to the PR #798 branch and tested for the standalone model in a P7 configuration. The required SDF can be found in the sandbox location show above. The control_p7 test case has been compiled in debug mode and is currently failing at

146: forrtl: error (65): floating invalid
146: Image              PC                Routine            Line        Source
146: fv3.exe            0000000008DDE69E  Unknown               Unknown  Unknown
146: libpthread-2.17.s  00002B063E89A630  Unknown               Unknown  Unknown
146: fv3.exe            00000000089D8BE2  bndlyr_                   311  BNDLYR.f
146: libiomp5.so        00002B063CFAAA43  __kmp_invoke_micr     Unknown  Unknown

This may indicate a problem with the SDF.

climbfuji · 2021-09-17T13:53:21Z

climbfuji
Sep 17, 2021
Maintainer

What is file BNDLYR.f? It is not in ccpp-physics and not in fv3atm.

1 reply

DeniseWorthen Sep 17, 2021
Maintainer Author

Hm....well, then that is a good question.

DeniseWorthen · 2021-09-17T13:53:30Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

@mdtoyNOAA

0 replies

junwang-noaa · 2021-09-17T13:57:10Z

junwang-noaa
Sep 17, 2021
Maintainer

Following Denise's testing, when ugwpv1 is completely turned off with following namelist changes, the decomposition test reproduce control.

<   ccpp_suite = 'FV3_GFS_v16_coupled_nsstNoahmp'
---
>   ccpp_suite = 'FV3_GFS_v16_coupled_nsstNoahmpUGWPv1'
187c187
<   do_tofd      = .true.
---
>   do_tofd      = .false.
193a194,201
>   gwd_opt      = 2
>   do_ugwp_v0   = .false.
>   do_ugwp_v1   = .true.
>   do_gsl_drag_ls_bl    = .true.
>   do_gsl_drag_ss       = .true.
>   do_gsl_drag_tofd     = .true.
>   do_ugwp_v1_orog_only = .false.
>   do_ugwp_v0_nst_only  = .false.
331a340
>   knob_ugwp_version = 1
339,342c348,358
<   knob_ugwp_dokdis  = 1
<   knob_ugwp_ndx4lh  = 1
<   knob_ugwp_version = 0
<   launch_level      = 54
---
>   knob_ugwp_dokdis  = 2
>   knob_ugwp_ndx4lh  = 4
>   knob_ugwp_palaunch = 275.0e2
>   knob_ugwp_nslope   = 1
>   knob_ugwp_lzmax    = 15.750e3
>   knob_ugwp_lzmin    = 0.75e3
>   knob_ugwp_lzstar   = 2.0e3
>   knob_ugwp_taumin   = 0.25e-3
>   knob_ugwp_tauamp   = 3.0e-3
>   knob_ugwp_lhmet    = 200.0e3
>   knob_ugwp_orosolv  = 'pss-1986'

0 replies

SMoorthi-emc · 2021-09-17T14:00:25Z

SMoorthi-emc
Sep 17, 2021

It is in UPP.

…

On Fri, Sep 17, 2021 at 9:53 AM Dom Heinzeller ***@***.***> wrote: What is file BNDLYR.f? It is not in ccpp-physics and not in fv3atm. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYUY5GCCOPVT5AMEJ7LUCNB57ANCNFSM5EHHMVTQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

1 reply

DeniseWorthen Sep 17, 2021
Maintainer Author

OK, I can turn off the post in the control_p7 test. Thanks.

DeniseWorthen · 2021-09-17T14:05:18Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

@SMoorthi-emc The right min_seaice for the uncoupled is 0.15? Or can I use 1.0e-6?

0 replies

SMoorthi-emc · 2021-09-17T14:06:41Z

SMoorthi-emc
Sep 17, 2021

Yes, 0.15.

…

On Fri, Sep 17, 2021 at 10:05 AM Denise Worthen ***@***.***> wrote: @SMoorthi-emc <https://github.com/SMoorthi-emc> The right min_seaice for the uncoupled is 0.15? Or can I use 1.0e-6? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#813 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYSI6V532HV6RAHXPQ3UCNDKVANCNFSM5EHHMVTQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

0 replies

DeniseWorthen · 2021-09-17T14:17:03Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

The control_p7 test is now running successfully in debug mode.

0 replies

DeniseWorthen · 2021-09-17T14:56:51Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

A sandbox for testing decomp in the standalone in P7 configuration using the FV3 in Mike Toy's PR branch
is at /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/DECOMP96_ATM/ . It was compiled using:

./compile.sh hera.intel '-DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_nsstNoahmpUGWPv1 -D32BIT=ON' '' YES NO 2>&1 | tee compile.p7.log

0 replies

DeniseWorthen · 2021-09-17T16:15:26Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

I tried two tests with the control_P7 test, one with all the settings as we use in P7 (changing layout only) and one turning off the tendencies (do_ugwp_v1 = .false., do_gsl_drag_ls_bl = .false., do_gsl_drag_ss = .false., do_gsl_drag_tofd = .false.).

The first test shows the following differences in the v-gwd_ave field for the sfcf000 files:

The second test (with zero tendencies) shows that the decomp test reproduces the base at forecast hour 12.

The run directories are /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/DECOMP96_ATM. The first test is tst0 and the second test is zerotend

1 reply

DeniseWorthen Sep 17, 2021
Maintainer Author

@mdtoyNOAA I believe I built your FV3 branch w/o the correct ccpp submodule (ie, w/o your fixes). I'll be repeating all my testing. Thanks.

mdtoyNOAA · 2021-09-17T17:32:19Z

mdtoyNOAA
Sep 17, 2021

Okay, let me know. I was literally about to e-mail you asking for confirmation that in an earlier sandbox test, you were able to get decomp reproducibility in atmosphere-only mode with my fixes.

…

On Sep 17, 2021, at 11:29 AM, Denise Worthen ***@***.***> wrote: @mdtoyNOAA <https://github.com/mdtoyNOAA> I believe I built your FV3 branch w/ the incorrect ccpp submodule (w/o your fixes). I'll be repeating all my testing. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#813 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARRVLIH4FXQ46UK323ZGFT3UCN3HHANCNFSM5EHHMVTQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

1 reply

DeniseWorthen Sep 17, 2021
Maintainer Author

I do apologize; after correctly building your FV3 branch and testing with the new control_P7 test, the decomp test does pass. The figures above clearly show some sort of decomp problem and I should have double checked my build.

I will verify that for the coupled model I am building correctly. If the coupled P7 test still fails the decomp test, we still need it resolved of course.

DeniseWorthen · 2021-09-17T20:54:20Z

DeniseWorthen
Sep 17, 2021
Maintainer Author

I've built the coupled model and tested it several ways. I get a consistent result with Jun who used the cpld_control_p7 test run as ATM only. The sfcf000.tile3.nc file fails to compare at a small number of points (between 1 and 7). All other tiles are B4B between the base and decomp tests. This executable contains the fix to GFS_surface_composites.F90 identified yesterday as part of the restart for the coupled model.

The fields which do not reproduce in the sfcf000.tile3.nc file are:

 RMS csulf                            4.0258E-04            NORMALIZED  1.3391E-06
 RMS csulftoa                         3.1259E-04            NORMALIZED  1.4131E-06
 RMS prate_ave                        1.4918E-16            NORMALIZED  5.4028E-12
 RMS prateb_ave                       1.4918E-16            NORMALIZED  5.4028E-12
 RMS tprcp                            1.3878E-17            NORMALIZED  6.9806E-13
 RMS ulwrf                            1.7119E-06            NORMALIZED  6.1510E-09
 RMS ulwrf_ave                        1.7119E-06            NORMALIZED  6.1510E-09
 RMS ulwrf_avetoa                     2.9441E-04            NORMALIZED  1.5392E-06

This is the csulf field, which has the 7 points different:

I've tested with frac_grid both on and off and I've tested with an executable built in debug mode. The results are consistent (only tile 3 differences).

My build directory is /scratch2/NCEPDEV/climate/Denise.Worthen/WORK/ufs_toy.

My sandbox is /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/DECOMP96

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decomp test fails for coupled model in P7 configuration #813

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 11 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Decomp test fails for coupled model in P7 configuration #813

DeniseWorthen Sep 17, 2021 Maintainer

Replies: 11 comments · 4 replies

climbfuji Sep 17, 2021 Maintainer

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

junwang-noaa Sep 17, 2021 Maintainer

SMoorthi-emc Sep 17, 2021

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

SMoorthi-emc Sep 17, 2021

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

mdtoyNOAA Sep 17, 2021

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen Sep 17, 2021 Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer

Replies: 11 comments 4 replies

climbfuji
Sep 17, 2021
Maintainer

DeniseWorthen Sep 17, 2021
Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer Author

junwang-noaa
Sep 17, 2021
Maintainer

SMoorthi-emc
Sep 17, 2021

DeniseWorthen Sep 17, 2021
Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer Author

SMoorthi-emc
Sep 17, 2021

DeniseWorthen
Sep 17, 2021
Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer Author

DeniseWorthen Sep 17, 2021
Maintainer Author

mdtoyNOAA
Sep 17, 2021

DeniseWorthen Sep 17, 2021
Maintainer Author

DeniseWorthen
Sep 17, 2021
Maintainer Author