Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Update ufs-weather-model hash and remove machine/modulefiles/WE2E test suites for decommissioned machines #1117

Merged
merged 13 commits into from
Sep 20, 2024

Conversation

MichaelLueken
Copy link
Collaborator

@MichaelLueken MichaelLueken commented Aug 20, 2024

DESCRIPTION OF CHANGES:

  • Update ufs-weather-model hash to a1143cc (August 12)
  • Remove machine files, modulefiles, and WE2E test suites for decommissioned machine, Cheyenne
  • Update build_jet_intel.lua modulefile to point to the /contrib spack-stack location, rather than /lfs4. At the end of September, /lfs4 is scheduled to be unmounted.
  • Update ush/machine/jet.yaml machine file to point to /lfs5 instead of /lfs4 for staged data on Jet. At the end of September, /lfs4 is schedules to be unmounted.
  • Updated ContainerQuickstart.rst and RunSRW.rst to note the new container location on Jet (/lfs5 instead of /lfs4) and the location of the staged data (/lfs5 instead of /lfs4).
  • Uncommented four WE2E tests that had previously failed in make_sfc_climo in the comprehensive WE2E suite for Derecho (comprehensive.derecho). Once this PR is merged, issue Failures encountered in make_sfc_climo on Derecho #947 can be closed.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

TESTS CONDUCTED:

  • derecho.intel - Fundamental and Comprehensive tests were run
  • gaea.intel - Fundamental and Comprehensive tests were run
  • hera.gnu - Fundamental tests were run
  • hera.intel - Fundamental, Comprehensive, AQM WE2E, and AQM sample configuration were all run
  • hercules.intel - Fundamental, Comprehensive, AQM WE2E, and AQM sample configuration were all run
  • jet.intel - Fundamental tests were run
  • orion.intel - Fundamental, Comprehensive, AQM WE2E, and AQM sample configuration were all run
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • My changes generate no new warnings
  • New and existing tests pass with my changes

@MichaelLueken
Copy link
Collaborator Author

The build_jet_intel.lua modulefile has been updated to point to the new /contrib spack-stack location, rather than /lfs4. The fundamental tests were run and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              12.69
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               9.81
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              19.45
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082  COMPLETE              51.10
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240828170  COMPLETE              34.17
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024082817055  COMPLETE              28.33
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             155.55

@MichaelLueken
Copy link
Collaborator Author

Following the update of the paths to staged data on Jet (from /lfs4 to /lfs5), in ush/machine/jet.yaml, the fundamental tests were run and all passed successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              12.47
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               9.59
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              31.83
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082  COMPLETE              51.75
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240829193  COMPLETE              34.39
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024082919352  COMPLETE              55.56
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             195.59

…re previously failing in Derecho comprehensive WE2E test suite
@MichaelLueken
Copy link
Collaborator Author

In issue #947, four WE2E tests were commented out in the Derecho comprehensive WE2E test suite. They were commented out due to errors generated in the make_sfc_climo task. This issue has been resolved. The full Derecho comprehensive WE2E test suite successfully passes now:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20240829065327                                            COMPLETE              79.41
2020_CAPE_20240829065330                                           COMPLETE              79.80
2019_hurricane_barry_20240829065331                                COMPLETE              74.65
2019_halloween_storm_20240829065333                                COMPLETE              75.47
2019_hurricane_lorenzo_20240829065334                              COMPLETE              75.23
2019_memorial_day_heat_wave_20240829065336                         COMPLETE              72.11
2020_denver_radiation_inversion_20240829065337                     COMPLETE              80.30
2020_easter_storm_20240829065338                                   COMPLETE              73.37
2020_jan_cold_blast_20240829065339                                 COMPLETE              81.24
community_20240829065340                                           COMPLETE              60.83
custom_ESGgrid_20240829065342                                      COMPLETE              42.27
custom_ESGgrid_Central_Asia_3km_20240829065343                     COMPLETE              71.60
custom_ESGgrid_IndianOcean_6km_20240829065345                      COMPLETE              43.83
custom_ESGgrid_NewZealand_3km_20240829065347                       COMPLETE             118.76
custom_ESGgrid_Peru_12km_20240829065349                            COMPLETE              63.86
custom_ESGgrid_SF_1p1km_20240829065350                             COMPLETE             333.39
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE              27.64
custom_GFDLgrid_20240829065354                                     COMPLETE              26.03
deactivate_tasks_20240829065356                                    COMPLETE               2.63
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE            1328.50
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240829065358              COMPLETE              35.91
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              33.38
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             680.42
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             268.22
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240829065405  COMPLETE             479.33
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065  COMPLETE              66.24
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              71.50
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024082906  COMPLETE              66.78
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              68.66
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE              21.50
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              41.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              41.81
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024082906  COMPLETE              46.32
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065  COMPLETE             138.12
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              56.76
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240829065422  COMPLETE              34.36
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE              27.77
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024082906542  COMPLETE              74.72
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202408290  COMPLETE              49.43
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202408  COMPLETE             467.33
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             627.11
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240829  COMPLETE             619.80
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240829065  COMPLETE             687.09
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             714.10
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              67.99
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240829  COMPLETE              60.68
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              62.20
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              37.20
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082  COMPLETE              62.01
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              29.26
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             540.44
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202408290  COMPLETE             816.60
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             846.79
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065445  COMPLETE             159.48
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202408  COMPLETE              80.10
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082906545  COMPLETE              60.43
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240829065  COMPLETE              44.75
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202408290654  COMPLETE              63.58
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE              27.98
MET_ensemble_verification_only_vx_20240829065457                   COMPLETE               3.22
MET_ensemble_verification_winter_wx_20240829065501                 COMPLETE             327.98
MET_verification_only_vx_20240829065504                            COMPLETE               0.93
pregen_grid_orog_sfc_climo_20240829065507                          COMPLETE              27.63
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240829065511               COMPLETE              24.99
specify_template_filenames_20240829065513                          COMPLETE              31.10
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE           11604.70

Uncommenting commented out WE2E tests from Derecho WE2E comprehensive suite now.

Copy link
Collaborator

@RatkoVasic-NOAA RatkoVasic-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and approved:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              11.44
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               9.47
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              29.37
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024090  COMPLETE              49.78
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240906045  COMPLETE              33.55
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024090604551  COMPLETE              55.83
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             189.44

Copy link

@rickgrubin-tomorrow rickgrubin-tomorrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

@rickgrubin-noaa rickgrubin-noaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MichaelLueken MichaelLueken added the run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests label Sep 19, 2024
@chan-hoo
Copy link
Collaborator

@MichaelLueken, did you test AQM with this PR? AQM is not compiled. I got the following error:

6580: Name in only-list does not exist or is not accessible.   [CLOSE_FILE]
use fms_mod,            only: close_file, write_version_number, stdlog, stdout
------------------------------^
compilation aborted for /scratch2/NAGAPE/epic/Chan-hoo.Jeon/test/ufs-srweather-app/sorc/ufs-weather-model/FV3/atmos_model.F90 (code 1)
make[5]: *** [FV3/CMakeFiles/fv3atm.dir/build.make:75: FV3/CMakeFiles/fv3atm.dir/atmos_model.F90.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:416: FV3/CMakeFiles/fv3atm.dir/all] Error 2
make[3]: *** [Makefile:136: all] Error 2
make[2]: *** [sorc/CMakeFiles/ufs-weather-model.dir/build.make:86: sorc/ufs-weather-model/src/ufs-weather-model-stamp/ufs-weather-model-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:893: sorc/CMakeFiles/ufs-weather-model.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Externals.cfg Show resolved Hide resolved
@MichaelLueken
Copy link
Collaborator Author

@chan-hoo -

The AQM WE2E test and sample configuration test were run on Hera prior to opening this PR.

This morning, I was able to successfully build the AQM executables using the a1143cc hash of the weather model:

(srw_app) [Michael.Lueken@hfe07 hera]$ ls exec/
chgres_cube   emcsfc_ice_blend	filter_topo  fvcom_to_FV3   global_cycle	inland	  make_hgrid	    nexus	 orog	   regional_esg_grid  shave	 ufs_srweather_app.settings  vcoord_gen
cpld_gridgen  emcsfc_snow2mdl	fregrid      gefs2lbc_para  global_equiv_resol	lakefrac  make_solo_mosaic  ocean_merge  orog_gsl  sfc_climo_gen      ufs_model  upp.x			     weight_gen

After successfully building, the aqm_grid_AQM_NA13km_suite_GFS_v16 WE2E test was launched and successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
aqm_grid_AQM_NA13km_suite_GFS_v16_20240920132716                   COMPLETE            5281.82
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            5281.82
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202311100000               make_grid                    66840814           SUCCEEDED                   0         1          38.0
202311100000               make_orog                    66840828           SUCCEEDED                   0         1         188.0
202311100000          make_sfc_climo                    66840940           SUCCEEDED                   0         1          66.0
202311100000           nexus_gfs_sfc                    66840815           SUCCEEDED                   0         1          35.0
202311100000       nexus_emission_00                    66840829           SUCCEEDED                   0         1         587.0
202311100000       nexus_emission_01                    66840830           SUCCEEDED                   0         1         588.0
202311100000       nexus_emission_02                    66840831           SUCCEEDED                   0         1         624.0
202311100000        nexus_post_split                    66841203           SUCCEEDED                   0         1         122.0
202311100000           fire_emission                    66840816           SUCCEEDED                   0         1          39.0
202311100000            point_source                    66840836           SUCCEEDED                   0         1         240.0
202311100000                aqm_lbcs                    66841182           SUCCEEDED                   0         1         101.0
202311100000           get_extrn_ics                    66840817           SUCCEEDED                   0         1          26.0
202311100000          get_extrn_lbcs                    66840818           SUCCEEDED                   0         1          28.0
202311100000         make_ics_mem000                    66840983           SUCCEEDED                   0         1         137.0
202311100000        make_lbcs_mem000                    66840984           SUCCEEDED                   0         1         330.0
202311100000         run_fcst_mem000                    66841298           SUCCEEDED                   0         1        2533.0
202311100000    run_post_mem000_f000                    66841720           SUCCEEDED                   0         1          43.0
202311100000    run_post_mem000_f001                    66841747           SUCCEEDED                   0         1          41.0
202311100000    run_post_mem000_f002                    66841918           SUCCEEDED                   0         1          42.0
202311100000    run_post_mem000_f003                    66841937           SUCCEEDED                   0         1          38.0
202311100000    run_post_mem000_f004                    66841981           SUCCEEDED                   0         1          36.0
202311100000    run_post_mem000_f005                    66842132           SUCCEEDED                   0         1          40.0
202311100000    run_post_mem000_f006                    66842220           SUCCEEDED                   0         1          38.0
202311100000    run_post_mem000_f007                    66842261           SUCCEEDED                   0         1          37.0
202311100000    run_post_mem000_f008                    66842310           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f009                    66842403           SUCCEEDED                   0         1          37.0
202311100000    run_post_mem000_f010                    66842440           SUCCEEDED                   0         1          37.0
202311100000    run_post_mem000_f011                    66842471           SUCCEEDED                   0         1          44.0
202311100000    run_post_mem000_f012                    66842545           SUCCEEDED                   0         1          38.0
202311100000    run_post_mem000_f013                    66842611           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f014                    66842668           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f015                    66842706           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f016                    66842868           SUCCEEDED                   0         1          36.0
202311100000    run_post_mem000_f017                    66842911           SUCCEEDED                   0         1          38.0
202311100000    run_post_mem000_f018                    66842927           SUCCEEDED                   0         1          35.0
202311100000    run_post_mem000_f019                    66843114           SUCCEEDED                   0         1          36.0
202311100000    run_post_mem000_f020                    66843176           SUCCEEDED                   0         1          43.0
202311100000    run_post_mem000_f021                    66843214           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f022                    66843329           SUCCEEDED                   0         1          39.0
202311100000    run_post_mem000_f023                    66843348           SUCCEEDED                   0         1          35.0
202311100000    run_post_mem000_f024                    66843349           SUCCEEDED                   0         1          45.0
202311100000    integration_test_mem000                    66843350           SUCCEEDED                   0         1          15.0
================================================================================================================================
202311110000           nexus_gfs_sfc                    66840819           SUCCEEDED                   0         1          35.0
202311110000       nexus_emission_00                    66840832           SUCCEEDED                   0         1         573.0
202311110000       nexus_emission_01                    66840833           SUCCEEDED                   0         1         567.0
202311110000       nexus_emission_02                    66840834           SUCCEEDED                   0         1         619.0
202311110000        nexus_post_split                    66841207           SUCCEEDED                   0         1         126.0
202311110000           fire_emission                    66840820           SUCCEEDED                   0         1          40.0
202311110000            point_source                    66840837           SUCCEEDED                   0         1         245.0
202311110000                 aqm_ics                    66843328           SUCCEEDED                   0         1         232.0
202311110000                aqm_lbcs                    66841186           SUCCEEDED                   0         1          96.0
202311110000           get_extrn_ics                    66840821           SUCCEEDED                   0         1          28.0
202311110000          get_extrn_lbcs                    66840822           SUCCEEDED                   0         1          26.0
202311110000         make_ics_mem000                    66840985           SUCCEEDED                   0         1         125.0
202311110000        make_lbcs_mem000                    66840986           SUCCEEDED                   0         1         334.0
202311110000         run_fcst_mem000                    66843551           SUCCEEDED                   0         1        2627.0
202311110000    run_post_mem000_f000                    66843954           SUCCEEDED                   0         1          38.0
202311110000    run_post_mem000_f001                    66844037           SUCCEEDED                   0         1          42.0
202311110000    run_post_mem000_f002                    66844086           SUCCEEDED                   0         1          42.0
202311110000    run_post_mem000_f003                    66844203           SUCCEEDED                   0         1          37.0
202311110000    run_post_mem000_f004                    66844248           SUCCEEDED                   0         1          42.0
202311110000    run_post_mem000_f005                    66844441           SUCCEEDED                   0         1          39.0
202311110000    run_post_mem000_f006                    66844597           SUCCEEDED                   0         1          38.0
202311110000    run_post_mem000_f007                    66844633           SUCCEEDED                   0         1          39.0
202311110000    run_post_mem000_f008                    66844661           SUCCEEDED                   0         1          40.0
202311110000    run_post_mem000_f009                    66844885           SUCCEEDED                   0         1          38.0
202311110000    run_post_mem000_f010                    66844937           SUCCEEDED                   0         1          35.0
202311110000    run_post_mem000_f011                    66844960           SUCCEEDED                   0         1          42.0
202311110000    run_post_mem000_f012                    66844970           SUCCEEDED                   0         1          44.0
202311110000    run_post_mem000_f013                    66845076           SUCCEEDED                   0         1          40.0
202311110000    run_post_mem000_f014                    66845136           SUCCEEDED                   0         1          38.0
202311110000    run_post_mem000_f015                    66845163           SUCCEEDED                   0         1          40.0
202311110000    run_post_mem000_f016                    66845243           SUCCEEDED                   0         1          39.0
202311110000    run_post_mem000_f017                    66845271           SUCCEEDED                   0         1          43.0
202311110000    run_post_mem000_f018                    66845344           SUCCEEDED                   0         1          39.0
202311110000    run_post_mem000_f019                    66845510           SUCCEEDED                   0         1          43.0
202311110000    run_post_mem000_f020                    66845583           SUCCEEDED                   0         1          40.0
202311110000    run_post_mem000_f021                    66845668           SUCCEEDED                   0         1          40.0
202311110000    run_post_mem000_f022                    66845796           SUCCEEDED                   0         1          38.0
202311110000    run_post_mem000_f023                    66845824           SUCCEEDED                   0         1          36.0
202311110000    run_post_mem000_f024                    66845825           SUCCEEDED                   0         1          36.0
202311110000    integration_test_mem000                    66845826           SUCCEEDED                   0         1          16.0

My build can be found /scratch1/NCEPDEV/stmp2/Michael.Lueken/ufs-srweather-app/hera and the expt_dirs can be found /scratch1/NCEPDEV/stmp2/Michael.Lueken/ufs-srweather-app/expt_dirs/aqm_grid_AQM_NA13km_suite_GFS_v16

Have you tried building AQM with the current develop?

Between the July 30 hash (b5a1976) and the August 12 hash (a1143cc), there weren't any changes to FMS, which is what appears to be causing the failure in your attempt to build AQM with my feature branch.

@chan-hoo
Copy link
Collaborator

@MichaelLueken, thanks for your re-test. I am not sure what was wrong. I'll try it again on a new terminal window.

@chan-hoo
Copy link
Collaborator

@MichaelLueken , it was compiled successfully on a new window. Approved.

@MichaelLueken
Copy link
Collaborator Author

Thanks, @chan-hoo!

@MichaelLueken
Copy link
Collaborator Author

The Jenkins tests successfully passed on Gaea, Hera Intel, Hera GNU, Hercules, and Orion.

The Jenkins tests weren't run on Derecho, as the machine node was still down following maintenance. The coverage WE2E tests were manually run on Derecho and the tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240919113453                      COMPLETE              35.89
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              74.21
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024091911345  COMPLETE              82.64
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240919  COMPLETE              53.71
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              25.37
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024091911350  COMPLETE              53.36
pregen_grid_orog_sfc_climo_20240919113503                          COMPLETE              23.55
specify_template_filenames_20240919113504                          COMPLETE              25.09
2019_hurricane_barry_20240919113505                                COMPLETE              63.99
2019_memorial_day_heat_wave_20240919113507                         COMPLETE              61.20
2020_denver_radiation_inversion_20240919113507                     COMPLETE              62.71
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             561.72

On Jet, the get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h WE2E test failed in make_lbcs due to OOM (this happens occasionally when kjet is used instead of vjet or xjet). The use of rocotorewind/rocotoboot allowed the test to pass successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
community_20240919212415                                           COMPLETE              23.19
custom_ESGgrid_20240919212417                                      COMPLETE              31.16
custom_ESGgrid_Great_Lakes_snow_8km_20240919212419                 COMPLETE              26.96
custom_GFDLgrid_20240919212421                                     COMPLETE              14.75
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202409  COMPLETE              14.82
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE             100.13
get_from_HPSS_ics_RAP_lbcs_RAP_20240919212424                      COMPLETE              19.69
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240919212425  COMPLETE             635.42
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              87.23
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               9.08
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             968.65
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1931.08

Preparing to merge now.

@MichaelLueken MichaelLueken merged commit 4212296 into ufs-community:develop Sep 20, 2024
2 checks passed
@MichaelLueken MichaelLueken deleted the feature/hash_update branch September 20, 2024 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failures encountered in make_sfc_climo on Derecho
7 participants