diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json index 056013dd1b..b5787cec6b 100644 --- a/.github/parm/use_case_groups.json +++ b/.github/parm/use_case_groups.json @@ -54,6 +54,11 @@ "index_list": "10-12", "run": false }, + { + "category": "short_range", + "index_list": "13", + "run": false + }, { "category": "data_assimilation", "index_list": "0-1", diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml index 762752c7e5..6f8c9b3d45 100644 --- a/.github/workflows/documentation.yml +++ b/.github/workflows/documentation.yml @@ -24,6 +24,7 @@ jobs: run: | python -m pip install --upgrade sphinx sphinx-gallery sphinx_rtd_theme python -m pip install python-dateutil requests Pillow + python -m pip install -r docs/requirements.txt - name: Build Documentation run: ./.github/jobs/build_documentation.sh - uses: actions/upload-artifact@v3 diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst index e31fb0c9ef..e6cffdac4a 100644 --- a/docs/Users_Guide/release-notes.rst +++ b/docs/Users_Guide/release-notes.rst @@ -34,206 +34,205 @@ describes the bugfix, enhancement, or new feature. METplus Version 5.1.0-beta1 Release Notes (2023-02-28) ------------------------------------------------------ -* Enhancements: + .. dropdown:: Enhancements - * Add support for multiple interp widths - (`#2049 `_) - * TCPairs - Add support for setting consensus.write_members - (`#2054 `_) + * Add support for multiple interp widths (`#2049 `_) + * TCPairs - Add support for setting consensus.write_members + (`#2054 `_) -* Bugfix: + .. dropdown:: Bugfix - * StatAnalysis - allow run once for each valid time - (`#2026 `_) + * StatAnalysis - allow run once for each valid time + (`#2026 `_) -* Documentation: + .. dropdown:: Documentation - * Update the METplus Components Python Requirements Documentation - (`#2016 `_) + * Update the METplus Components Python Requirements Documentation + (`#2016 `_) -* Internal: + .. dropdown:: Internal - * Improve use case testing - (`#685 `_) - * Update conda environments to use 3.10 for automated use case tests - (`#2005 `_) - * Add modulefiles to the repository - (`#2015 `_) - * **Upgrade to using Python 3.10.4** - (`#2022 `_) - * Add 'License.txt' to the METplus repo - (`#2058 `_) + * Improve use case testing + (`#685 `_) + * Update conda environments to use 3.10 for automated use case tests + (`#2005 `_) + * Add modulefiles to the repository + (`#2015 `_) + * **Upgrade to using Python 3.10.4** + (`#2022 `_) + * Add 'License.txt' to the METplus repo + (`#2058 `_) METplus Version 5.0.0 Release Notes (2022-12-09) ------------------------------------------------ -.. warning:: **MAJOR CHANGES**: - - * The LOOP_ORDER config variable was removed. The variable set in a user's - config file will be ignored in favor of executing the logic that - corresponds to *LOOP_ORDER = processes*, where all times are processed for - the first item in the PROCESS_LIST, then all times are processed for the - second item in the PROCESS_LIST, etc. This may change the order that - commands are executed in a use case, but it should not change the results. - * The METplus Dockerfile was moved to internal/scripts/docker. - It was previously found in scripts/docker. - * Use cases that include **EnsembleStat** wrapper will require config file - updates. See :ref:`upgrade-instructions`. - * The default value of :term:`SCRUB_STAGING_DIR` is now *True*. - This means some intermediate files that are auto-generated by METplus such - as file lists and uncompressed files will automatically be removed unless - this option is set by the user. - These files are typically only used to debug unexpected issues. - * The default value of :term:`METPLUS_CONF` now includes the - :term:`LOG_TIMESTAMP` so each METplus run will generate a unique final - config file, e.g. metplus_final.conf.20220921121733. - - -* Enhancements: - - * **Enhance MODE wrapper to support multi-variate MODE** - (`#1585 `_) - * **Allow FCST_IS_PROB variable setting specific to tool - (FCST__IS_PROB)** - (`#1586 `_) - * **Enhance climatology field settings to be consistent with fcst/obs field** - (`#1599 `_) - * Update Hovmoeller Use case to use updated Hovmoeller plotting - (`#1650 `_) - * **Update the EnsembleStat wrapper and use case examples to remove - ensemble post processing logic** - (`#1816 `_) - * Enhance logic to consistently create directories - (`#1657 `_) - * Create checksum for released code - (`#262 `_) - * Add the user ID to the log output at beginning and end of each - METplus wrappers run - (`dtcenter/METplus-Internal#20 `_) - * Update logic to name final conf and intermediate files with a unique - identifier - (`dtcenter/METplus-Internal#32 `_) - * Change default logging time information - (`dtcenter/METplus-Internal#34 `_) - * **Remove LOOP_ORDER config variable** - (`#1687 `_) - * **Add unique identifier for each METplus run to configuration** - (`#1829 `_) - * StatAnalysis - Support setting multiple jobs - (`#1842 `_) - * StatAnalysis - Set MET verbosity - (`#1772 `_) - * StatAnalysis - Support using both init/valid variables in - string substitution - (`#1861 `_) - * StatAnalysis - Allow filename template tags in jobs - (`#1862 `_) - * StatAnalysis - Support looping over groups of list items - (`#1870 `_) - * StatAnalysis - Allow processing of time ranges other than daily - (`#1871 `_) - * StatAnalysis - Add support for using a custom loop list - (`#1893 `_) - * Remove MakePlots wrapper - (`#1843 `_) - * Add support in EnsembleStat wrapper for setting -ens_mean - command line argument - (`#1569 `_) - * Enhance METplus to have better signal handling for shutdown events - (`dtcenter/METplus-Internal#27 `_) - * TCPairs and TCStat - add support for new config options and - command line arguments - (`#1898 `_) - * Enhance the GridStat and PointStat wrappers to handle the - addition of SEEPS - (`#1953 `_) - * SeriesAnalysis - add support for setting mask dictionary - (`#1926 `_) - * Update Python requirement to 3.8.6 - (`#1566 `_) - * Enhance StatAnalysis wrapper to support now and today - (`#1669 `_) - * **Clean up use case configuration files** - (`#1402 `_) - * Add support for creating multiple input datasets - (`#1694 `_) - -* Bugfixes: - - * PCPCombine - custom loop list does not work for subtract method - (`#1884 `_) - * Set level properly in filename template for EnsembleStat forecast input - (`#1910 `_) - * Prevent duplicate observation files using a file window if - compressed equivalent files exist in same directory - (`#1939 `_) - * Allow NA value for _CLIMO_[MEAN/STDEV]_HOUR_INTERVAL - (`#1787 `_) - * Reconcile setting of METPLOTPY_BASE for use cases - (`#1713 `_) - * Add support for the {custom} loop string in the MODEL config variable - (`#1382 `_) - * Fix PCPCombine extra options removal of semi-colon - (`#1534 `_) - * Fix reset of arguments for some wrappers - (i.e. GenEnsProd) after each run - (`#1555 `_) - * Enhance METDbLoad Wrapper to find MODE .txt files - (`#1608 `_) - * Add missing brackets around list variable values for StatAnalysis wrapper - (`#1641 `_) - * Allow NA value for _CLIMO_[MEAN/STDEV]_DAY_INTERVAL - (`#1653 `_) - -* New Wrappers: - - * PlotPointObs - (`#1489 `_) - -* New Use Cases: - - * PANDA-C use cases - (`#1686 `_) - * MJO-ENSO diagnostics - (`#1330 `_) - * Probability of Exceedence for 85th percentile temperatures - (`#1808 `_) - * FV3 Physics Tendency plotting via METplotpy - (`#1852 `_) - * StatAnalysis Python Embedding using IODA v2.0 - (`#1453 `_) - * StatAnalysis Python Embedding to read native grid (u-grid) - (`#1561 `_) - -* Documentation: - - * Update documentation to include instructions - to disable UserScript wrapper - (`dtcenter/METplus-Internal#33 `_) - -* Internal: - - * Organize utility scripts used by multiple wrappers - (`#344 `_) - * Fix GitHub Actions warnings - update the version of actions - and replace set-output - (`#1863 `_) - * Update diff logic to handle CSV files that have rounding differences - (`#1865 `_) - * Add unit tests for expected failure - (`dtcenter/METplus-Internal#24 `_) - * Add instructions in Release Guide for "Recreate an Existing Release" - (`#1746 `_) - * Add modulefiles used for installations on various machines - (`#1749 `_) - * Document GitHub Discussions procedure for the Contributor's Guide - (`#1159 `_) - * Create a METplus "Release Guide" describing how to build - releases for the METplus components - (`#673 `_) - * Update documentation about viewing RTD URLs on branches - (`#1512 `_) + .. warning:: **MAJOR CHANGES**: + + * The LOOP_ORDER config variable was removed. The variable set in a user's + config file will be ignored in favor of executing the logic that + corresponds to *LOOP_ORDER = processes*, where all times are processed for + the first item in the PROCESS_LIST, then all times are processed for the + second item in the PROCESS_LIST, etc. This may change the order that + commands are executed in a use case, but it should not change the results. + * The METplus Dockerfile was moved to internal/scripts/docker. + It was previously found in scripts/docker. + * Use cases that include **EnsembleStat** wrapper will require config file + updates. See :ref:`upgrade-instructions`. + * The default value of :term:`SCRUB_STAGING_DIR` is now *True*. + This means some intermediate files that are auto-generated by METplus such + as file lists and uncompressed files will automatically be removed unless + this option is set by the user. + These files are typically only used to debug unexpected issues. + * The default value of :term:`METPLUS_CONF` now includes the + :term:`LOG_TIMESTAMP` so each METplus run will generate a unique final + config file, e.g. metplus_final.conf.20220921121733. + + + .. dropdown:: Enhancements + + * **Enhance MODE wrapper to support multi-variate MODE** + (`#1585 `_) + * **Allow FCST_IS_PROB variable setting specific to tool + (FCST__IS_PROB)** + (`#1586 `_) + * **Enhance climatology field settings to be consistent with fcst/obs field** + (`#1599 `_) + * Update Hovmoeller Use case to use updated Hovmoeller plotting + (`#1650 `_) + * **Update the EnsembleStat wrapper and use case examples to remove + ensemble post processing logic** + (`#1816 `_) + * Enhance logic to consistently create directories + (`#1657 `_) + * Create checksum for released code + (`#262 `_) + * Add the user ID to the log output at beginning and end of each + METplus wrappers run + (`dtcenter/METplus-Internal#20 `_) + * Update logic to name final conf and intermediate files with a unique + identifier + (`dtcenter/METplus-Internal#32 `_) + * Change default logging time information + (`dtcenter/METplus-Internal#34 `_) + * **Remove LOOP_ORDER config variable** + (`#1687 `_) + * **Add unique identifier for each METplus run to configuration** + (`#1829 `_) + * StatAnalysis - Support setting multiple jobs + (`#1842 `_) + * StatAnalysis - Set MET verbosity + (`#1772 `_) + * StatAnalysis - Support using both init/valid variables in + string substitution + (`#1861 `_) + * StatAnalysis - Allow filename template tags in jobs + (`#1862 `_) + * StatAnalysis - Support looping over groups of list items + (`#1870 `_) + * StatAnalysis - Allow processing of time ranges other than daily + (`#1871 `_) + * StatAnalysis - Add support for using a custom loop list + (`#1893 `_) + * Remove MakePlots wrapper + (`#1843 `_) + * Add support in EnsembleStat wrapper for setting -ens_mean + command line argument + (`#1569 `_) + * Enhance METplus to have better signal handling for shutdown events + (`dtcenter/METplus-Internal#27 `_) + * TCPairs and TCStat - add support for new config options and + command line arguments + (`#1898 `_) + * Enhance the GridStat and PointStat wrappers to handle the + addition of SEEPS + (`#1953 `_) + * SeriesAnalysis - add support for setting mask dictionary + (`#1926 `_) + * Update Python requirement to 3.8.6 + (`#1566 `_) + * Enhance StatAnalysis wrapper to support now and today + (`#1669 `_) + * **Clean up use case configuration files** + (`#1402 `_) + * Add support for creating multiple input datasets + (`#1694 `_) + + .. dropdown:: Bugfixes + + * PCPCombine - custom loop list does not work for subtract method + (`#1884 `_) + * Set level properly in filename template for EnsembleStat forecast input + (`#1910 `_) + * Prevent duplicate observation files using a file window if + compressed equivalent files exist in same directory + (`#1939 `_) + * Allow NA value for _CLIMO_[MEAN/STDEV]_HOUR_INTERVAL + (`#1787 `_) + * Reconcile setting of METPLOTPY_BASE for use cases + (`#1713 `_) + * Add support for the {custom} loop string in the MODEL config variable + (`#1382 `_) + * Fix PCPCombine extra options removal of semi-colon + (`#1534 `_) + * Fix reset of arguments for some wrappers + (i.e. GenEnsProd) after each run + (`#1555 `_) + * Enhance METDbLoad Wrapper to find MODE .txt files + (`#1608 `_) + * Add missing brackets around list variable values for StatAnalysis wrapper + (`#1641 `_) + * Allow NA value for _CLIMO_[MEAN/STDEV]_DAY_INTERVAL + (`#1653 `_) + + .. dropdown:: New Wrappers + + * PlotPointObs + (`#1489 `_) + + .. dropdown:: New Use Cases + + * PANDA-C use cases + (`#1686 `_) + * MJO-ENSO diagnostics + (`#1330 `_) + * Probability of Exceedence for 85th percentile temperatures + (`#1808 `_) + * FV3 Physics Tendency plotting via METplotpy + (`#1852 `_) + * StatAnalysis Python Embedding using IODA v2.0 + (`#1453 `_) + * StatAnalysis Python Embedding to read native grid (u-grid) + (`#1561 `_) + + .. dropdown:: Documentation + + * Update documentation to include instructions + to disable UserScript wrapper + (`dtcenter/METplus-Internal#33 `_) + + .. dropdown:: Internal + + * Organize utility scripts used by multiple wrappers + (`#344 `_) + * Fix GitHub Actions warnings - update the version of actions + and replace set-output + (`#1863 `_) + * Update diff logic to handle CSV files that have rounding differences + (`#1865 `_) + * Add unit tests for expected failure + (`dtcenter/METplus-Internal#24 `_) + * Add instructions in Release Guide for "Recreate an Existing Release" + (`#1746 `_) + * Add modulefiles used for installations on various machines + (`#1749 `_) + * Document GitHub Discussions procedure for the Contributor's Guide + (`#1159 `_) + * Create a METplus "Release Guide" describing how to build + releases for the METplus components + (`#673 `_) + * Update documentation about viewing RTD URLs on branches + (`#1512 `_) .. _upgrade-instructions: diff --git a/docs/Users_Guide/statistics_list.rst b/docs/Users_Guide/statistics_list.rst index 320c570334..537c3063fc 100644 --- a/docs/Users_Guide/statistics_list.rst +++ b/docs/Users_Guide/statistics_list.rst @@ -2295,7 +2295,7 @@ ____________________ Use Case - n/a * - Spatial distance between :raw-html:`
` - (𝑥,𝑦)(x,y) coordinates of :raw-html:`
` + :math:`(x,y)` coordinates of :raw-html:`
` object spacetime centroid - SPACE :raw-html:`
` _CENTROID :raw-html:`
` diff --git a/docs/_static/short_range-MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.png b/docs/_static/short_range-MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.png new file mode 100644 index 0000000000..2c45df04c2 Binary files /dev/null and b/docs/_static/short_range-MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.png differ diff --git a/docs/conf.py b/docs/conf.py index 77c12a540a..4c920764c4 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -113,6 +113,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx_gallery.gen_gallery', + 'sphinx_design', ] # settings for ReadTheDocs PDF creation diff --git a/docs/requirements.txt b/docs/requirements.txt index a45484cc64..9e01ed25f1 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,3 +1,4 @@ sphinx-gallery==0.11.1 -sphinx==6.0.0 -sphinx-rtd-theme==1.2.0rc2 +sphinx==5.3.0 +sphinx-rtd-theme==1.2.0 +sphinx-design==0.3.0 diff --git a/docs/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.py b/docs/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.py index c638b039d8..e7e0407072 100644 --- a/docs/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.py +++ b/docs/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.py @@ -13,12 +13,14 @@ # -------------------- # # Run PCPCombine, GenEnsProd, and RegridDataPlane tools to create surrogate severe probability -# forecasts (SSPFs) for a given date. SSPFs are a severe weather forecasting tool and is a techniqu -# used by the Storm Prediction Center (SPC) as well as others. SSPFs are based on updraft helicity -# (UH; UH = ∫z0 to zt (ω * ζ) dz) since certain thresholds of UH have been shown as good proxies for# severe weather. SSPFs can be thought of as the perfect model forecast. They are derived as follows: +# forecasts (SSPFs) for a given date. SSPFs are a severe weather forecasting tool and is a technique +# used by the Storm Prediction Center (SPC) as well as others. SSPFs are based on updraft helicity +# (UH; :math:`\text{UH} = \int_{z_0}^{z_t} ( \omega * \zeta ) dz`) since certain thresholds of UH +# have been shown as good proxies for severe weather. SSPFs can be thought of as the perfect model +# forecast. They are derived as follows: # # 1. Regrid the maximum UH value over the 2-5km layer at each grid point to the NCEP 211 grid (dx = ~80km). -# 2. Create a binary mask of points that meet a given threshold of UH) +# 2. Create a binary mask of points that meet a given threshold of UH. # 3. Convert the binary mask into a probability field by applying a Gaussian filter. # # For more information, please reference Sobash et al. 2011 (https://journals.ametsoc.org/doi/full/10.1175/WAF-D-10-05046.1). diff --git a/docs/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.py b/docs/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.py new file mode 100644 index 0000000000..db6b138f37 --- /dev/null +++ b/docs/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.py @@ -0,0 +1,196 @@ +""" +MODE: Multivariate +========================================================================= + +model_applications/ +short_range/ +MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf + +""" +############################################################################## +# Scientific Objective +# -------------------- +# +# This use case demonstrates how to run Multivariate MODE to identify complex +# objects from two or more fields, defined by a logical expression. This use +# case identifies blizzard-like objects defined by: 1) the presence of snow +# precipitation type, 2) 10-m winds > 20 mph, and 3) visibility < 1/2 mile. +# The use of multivariate MODE is well-suited to assess the structure and +# placement of complex high-impact events such as blizzard conditions and heavy +# snow bands. Output from this use-case consists of the MODE forecast and observation +# super objects and the MODE ASCII, NetCDF, and PostScript files. +# + +############################################################################## +# Datasets +# -------- +# +# **Forecast dataset:** 1-hour HRRR in grib2 +# +# **Observation dataset:** MRMS and HRRR analysis in grib2 +# +# The forecast and observation fields are only a subset of the full domain in +# order for a faster run-time of Multivariate MODE. An example command using +# wgrib2 to create the HRRR subdomain is:: +# +# wgrib2 infile.grib2 -new_grid_winds earth -new_grid lambert:262.5:38.5:38.5:38.5 -83.0:400:3000 37.0:400:3000 outfile.grib2 +# +# **Location:** All of the input data required for this use case can be found +# in the *short_range* sample data tarball. +# Navigate to `METplus Releases `_ +# and download sample data for the appropriate release. +# +# This tarball should be unpacked into the directory that you will set the +# value of INPUT_BASE. See :ref:`running-metplus` for more information. + + +############################################################################## +# METplus Components +# ------------------ +# +# This use case runs MODE using multiple variables to output the super objects +# based on a user-defined logical expression. Currently, the initial multivariate +# MODE run only outputs the super objects and additional steps are required to +# produce the statistical output. GenVxMask is run on a field(s) of interest +# using the super objects to mask the field(s). Finally, MODE is run a second +# time on the super-object-masked field(s) to output attribute statistics for +# the field(s). +# +# **Note:** The second MODE run can also be run directly on the super objects if +# field-specific statistics, such as intensity, is not desired. +# + +############################################################################## +# METplus Workflow +# ---------------- +# +# The following tools are used for each run time: +# +# MODE(mv), GenVxMask(fcst_super), GenVxMask(obs_super), MODE(super) +# +# Where the first instance of MODE runs over multiple variables to identify +# super objects for the forecast and observation, GenVxMask masks the raw input +# field(s) using the super objects, and the second instance of MODE is run +# traditionally to compare the masked forecast and observed super objects and +# and provide statistics. +# +# This example runs a single forecast hour. +# +# | **Initialization:** 2021020100 +# | **Forecast lead:** 21 +# + +############################################################################## +# METplus Configuration +# --------------------- +# +# METplus first loads all of the configuration files found in parm/metplus_config, +# then it loads any configuration files passed to METplus via the command line: +# parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf + +############################################################################## +# MET Configuration +# --------------------- +# +# METplus sets environment variables based on user settings in the METplus configuration file. +# See :ref:`How METplus controls MET config file settings` for more details. +# +# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!** +# +# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to: +# :ref:`Overriding Unsupported MET config file settings` +# +# .. note:: See the :ref:`MODE MET Configuration` section of the User's Guide for more information on the environment variables used in the file below: +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/met_config/MODEConfig_wrapped + +############################################################################## +# Running METplus +# --------------- +# +# Pass the use case configuration file to the run_metplus.py script +# along with any user-specific system configuration files if desired:: +# +# run_metplus.py /path/to/METplus/parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf /path/to/user_system.conf +# +# See :ref:`running-metplus` for more information. + +############################################################################## +# Expected Output +# --------------- +# +# A successful run will output the following both to the screen and to the logfile:: +# +# INFO: METplus has successfully finished running. +# +# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated. +# Output for this use case will be found in OUTPUT_BASE for the various MET tools +# and will contain the following files: +# +# **mode/2021020100/f21** +# +# Multivariate output - first instance +# +# Precipitation type = snow +# +# * 00/mode_210000L_20210201_210000V_000000A_cts.txt +# * 00/mode_210000L_20210201_210000V_000000A_obj.nc +# * 00/mode_210000L_20210201_210000V_000000A_obj.txt +# * 00/mode_210000L_20210201_210000V_000000A.ps +# +# Visibility +# +# * 01/mode_210000L_20210201_210000V_000000A_cts.txt +# * 01/mode_210000L_20210201_210000V_000000A_obj.nc +# * 01/mode_210000L_20210201_210000V_000000A_obj.txt +# * 01/mode_210000L_20210201_210000V_000000A.ps +# +# 10-m Winds +# +# * 02/mode_210000L_20210201_210000V_000000A_cts.txt +# * 02/mode_210000L_20210201_210000V_000000A_obj.nc +# * 02/mode_210000L_20210201_210000V_000000A_obj.txt +# * 02/mode_210000L_20210201_210000V_000000A.ps +# +# Super Objects +# +# * f_super.nc +# * o_super.nc +# +# MODE 10-m wind super object output - second instance +# +# * mode_HRRR_vs_ANALYSIS_WIND_super_Z10_210000L_20210201_210000V_000000A_cts.txt +# * mode_HRRR_vs_ANALYSIS_WIND_super_Z10_210000L_20210201_210000V_000000A_obj.nc +# * mode_HRRR_vs_ANALYSIS_WIND_super_Z10_210000L_20210201_210000V_000000A_obj.txt +# * mode_HRRR_vs_ANALYSIS_WIND_super_Z10_210000L_20210201_210000V_000000A.ps +# +# **gen_vx_mask/2021020100** +# +# * fcst_wind_super_2021020100_f21.nc +# * obs_wind_super_2021020121.nc + +############################################################################## +# Keywords +# -------- +# +# .. note:: +# +# * MODEToolUseCase +# * GenVxMaskToolUseCase +# * ShortRangeAppUseCase +# * GRIB2FileUseCase +# * RegriddingInToolUseCase +# * NOAAWPCOrgUseCase +# * NCAROrgUseCase +# * DiagnosticsUseCase +# +# +# Navigate to the :ref:`quick-search` page to discover other similar use cases. +# +# +# sphinx_gallery_thumbnail_path = '_static/short_range-MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.png' +# diff --git a/internal/scripts/installation/modulefiles/5.0.0.lua_wcoss2 b/internal/scripts/installation/modulefiles/5.0.0.lua_wcoss2 deleted file mode 100644 index 2f5d5cf7c5..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0.lua_wcoss2 +++ /dev/null @@ -1,26 +0,0 @@ -help([[ -]]) - -local pkgName = myModuleName() -local pkgVersion = myModuleVersion() -local pkgNameVer = myModuleFullName() - -local hierA = hierarchyA(pkgNameVer,1) -local compNameVer = hierA[1] - -conflict(pkgName) - -local opt = os.getenv("HPC_OPT") or os.getenv("OPT") or "/opt/modules" - -local base = pathJoin(opt,compNameVer,pkgName,pkgVersion) - -prepend_path("PATH", pathJoin(base,"ush")) - -setenv("METPLUS_ROOT", base) -setenv("METPLUS_VERSION", pkgVersion) -setenv("METPLUS_PATH", base) - -whatis("Name: ".. pkgName) -whatis("Version: " .. pkgVersion) -whatis("Category: application") -whatis("Description: Model Evaluation Tools Plus (METplus)") diff --git a/internal/scripts/installation/modulefiles/5.0.0_casper b/internal/scripts/installation/modulefiles/5.0.0_casper deleted file mode 100644 index ed1702f174..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0_casper +++ /dev/null @@ -1,18 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0.0. - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module use /glade/p/ral/jntp/MET/MET_releases/casper/modulefiles -module load met/11.0.0 -module load nco -module load grib-bins/1.3 -module load R - -prepend-path PATH /glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin -setenv METPLUS_PATH /glade/p/ral/jntp/MET/METplus/casper/METplus-5.0.0 -prepend-path PATH /glade/p/ral/jntp/MET/METplus/casper/METplus-5.0.0/ush diff --git a/internal/scripts/installation/modulefiles/5.0.0_cheyenne b/internal/scripts/installation/modulefiles/5.0.0_cheyenne deleted file mode 100644 index 6a050683f9..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0_cheyenne +++ /dev/null @@ -1,18 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0. - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module use /glade/p/ral/jntp/MET/MET_releases/modulefiles -module load met/11.0.0 -module load nco -module load grib-bins/1.3 -module load R - -setenv METPLUS_PATH /glade/p/ral/jntp/MET/METplus/METplus-5.0.0 -prepend-path PATH /glade/p/ral/jntp/MET/METplus/METplus-5.0.0/ush:/glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin - diff --git a/internal/scripts/installation/modulefiles/5.0.0_frontera b/internal/scripts/installation/modulefiles/5.0.0_frontera deleted file mode 100644 index d215a82532..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0_frontera +++ /dev/null @@ -1,22 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0.0 - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module load intel/19.1.1 -module load nco/4.9.7 -module load Rstats/4.0.3 -module load met/11.0.0 -setenv METPLUS_PATH /work2/06612/tg859120/frontera/METplus/METplus-5.0.0 - -# Path to wgrib -prepend-path PATH /work2/08291/taosun/HPC-STACK/hpc-module/intel-18.0.2/grib_util/1.2.2/bin - -# Path to wgrib2 -prepend-path PATH /work2/08291/taosun/HPC-STACK/hpc-module/intel-18.0.2/wgrib2/2.0.8/bin - -prepend-path PATH /work2/06612/tg859120/frontera/METplus/METplus-5.0.0/ush:/work2/06612/tg859120/frontera/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin diff --git a/internal/scripts/installation/modulefiles/5.0.0_gaea b/internal/scripts/installation/modulefiles/5.0.0_gaea deleted file mode 100644 index fdd6fe40cd..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0_gaea +++ /dev/null @@ -1,18 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0. - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module load intel/19.0.5.281 -module use -a /usw/met/modulefiles -module load met/11.0.0 -module load nco -module load wgrib -module load wgrib2 - -setenv METPLUS_PATH /usw/met/METplus/METplus-5.0.0 -prepend-path PATH /usw/met/METplus/METplus-5.0.0/ush:/lustre/f2/dev/esrl/Julie.Prestopnik/projects/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin diff --git a/internal/scripts/installation/modulefiles/5.0.0_orion b/internal/scripts/installation/modulefiles/5.0.0_orion deleted file mode 100644 index 8d15f0352c..0000000000 --- a/internal/scripts/installation/modulefiles/5.0.0_orion +++ /dev/null @@ -1,17 +0,0 @@ -#%Module###################################################################### -## -## METplus -## -proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0.0 - *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" -} - -module load contrib -module load intel/2020.2 -module load met/11.0.0 -module load nco/4.8.1 -module load wgrib/2.0.8 -prepend-path PATH /work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin -setenv METPLUS_PATH /apps/contrib/MET/METplus/METplus-5.0.0 -prepend-path PATH /apps/contrib/MET/METplus/METplus-5.0.0/ush diff --git a/internal/scripts/installation/modulefiles/5.0.0_hera b/internal/scripts/installation/modulefiles/5.1.0_hera similarity index 66% rename from internal/scripts/installation/modulefiles/5.0.0_hera rename to internal/scripts/installation/modulefiles/5.1.0_hera index 3faac54eb8..ff8e845589 100644 --- a/internal/scripts/installation/modulefiles/5.0.0_hera +++ b/internal/scripts/installation/modulefiles/5.1.0_hera @@ -3,20 +3,18 @@ ## METplus ## proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus-5.0. + puts stderr "Sets up the paths and environment variables to use the METplus-5.1.0. *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" } prereq intel -module use -a /contrib/anaconda/modulefiles -module load anaconda/latest -prepend-path PATH /scratch1/BMC/dtc/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin +prepend-path PATH /scratch1/BMC/dtc/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin module use -a /contrib/met/modulefiles -module load met/11.0.0 +module load met/11.1.0 module load nco module load wgrib module load wgrib2 module load R -setenv METPLUS_PATH /contrib/METplus/METplus-5.0.0 -prepend-path PATH /contrib/METplus/METplus-5.0.0/ush +setenv METPLUS_PATH /contrib/METplus/METplus-5.1.0 +prepend-path PATH /contrib/METplus/METplus-5.1.0/ush diff --git a/internal/scripts/installation/modulefiles/5.0.0_jet b/internal/scripts/installation/modulefiles/5.1.0_jet similarity index 61% rename from internal/scripts/installation/modulefiles/5.0.0_jet rename to internal/scripts/installation/modulefiles/5.1.0_jet index e0e1df41fc..c2db60aeb3 100644 --- a/internal/scripts/installation/modulefiles/5.0.0_jet +++ b/internal/scripts/installation/modulefiles/5.1.0_jet @@ -3,7 +3,7 @@ ## Model Evaluation Tools ## proc ModulesHelp { } { - puts stderr "Sets up the paths and environment variables to use the METplus v5.0.0 + puts stderr "Sets up the paths and environment variables to use the METplus v5.1.0 *** For help see the official MET webpage at http://www.dtcenter.org/met/users ***" } @@ -13,9 +13,8 @@ prereq hdf5/1.10.5 prereq nco/4.9.1 prereq wgrib/1.8.1.0b prereq wgrib2/2.0.8 -prereq R/4.0.2 -prereq met/11.0.0 - -setenv METPLUS_PATH /contrib/met/METplus/METplus-5.0.0 -prepend-path PATH /contrib/met/METplus/METplus-5.0.0/ush:/mnt/lfs1/HFIP/dtc-hurr/METplus/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin +prereq R/4.0.2 +prereq met/11.1.0 +setenv METPLUS_PATH /contrib/met/METplus/METplus-5.1.0 +prepend-path PATH /contrib/met/METplus/METplus-5.1.0/ush:/mnt/lfs1/HFIP/dtc-hurr/METplus/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin diff --git a/internal/tests/pytests/util/config_metplus/test_config_metplus.py b/internal/tests/pytests/util/config_metplus/test_config_metplus.py index 71fe75138b..5b374fe011 100644 --- a/internal/tests/pytests/util/config_metplus/test_config_metplus.py +++ b/internal/tests/pytests/util/config_metplus/test_config_metplus.py @@ -103,18 +103,18 @@ def test_find_indices_in_config_section(metplus_config, regex, index, @pytest.mark.parametrize( 'config_var_name, expected_indices, set_met_tool', [ - ('FCST_GRID_STAT_VAR1_NAME', ['1'], True), - ('FCST_GRID_STAT_VAR2_INPUT_FIELD_NAME', ['2'], True), - ('FCST_GRID_STAT_VAR3_FIELD_NAME', ['3'], True), - ('BOTH_GRID_STAT_VAR4_NAME', ['4'], True), - ('BOTH_GRID_STAT_VAR5_INPUT_FIELD_NAME', ['5'], True), - ('BOTH_GRID_STAT_VAR6_FIELD_NAME', ['6'], True), - ('FCST_VAR7_NAME', ['7'], False), - ('FCST_VAR8_INPUT_FIELD_NAME', ['8'], False), - ('FCST_VAR9_FIELD_NAME', ['9'], False), - ('BOTH_VAR10_NAME', ['10'], False), - ('BOTH_VAR11_INPUT_FIELD_NAME', ['11'], False), - ('BOTH_VAR12_FIELD_NAME', ['12'], False), + ('FCST_GRID_STAT_VAR1_NAME', [1], True), + ('FCST_GRID_STAT_VAR2_INPUT_FIELD_NAME', [2], True), + ('FCST_GRID_STAT_VAR3_FIELD_NAME', [3], True), + ('BOTH_GRID_STAT_VAR4_NAME', [4], True), + ('BOTH_GRID_STAT_VAR5_INPUT_FIELD_NAME', [5], True), + ('BOTH_GRID_STAT_VAR6_FIELD_NAME', [6], True), + ('FCST_VAR7_NAME', [7], False), + ('FCST_VAR8_INPUT_FIELD_NAME', [8], False), + ('FCST_VAR9_FIELD_NAME', [9], False), + ('BOTH_VAR10_NAME', [10], False), + ('BOTH_VAR11_INPUT_FIELD_NAME', [11], False), + ('BOTH_VAR12_FIELD_NAME', [12], False), ] ) @pytest.mark.util @@ -126,12 +126,14 @@ def test_find_var_indices_fcst(metplus_config, data_types = ['FCST'] config.set('config', config_var_name, "NAME1") met_tool = 'grid_stat' if set_met_tool else None - var_name_indices = config_metplus._find_var_name_indices(config, - data_types=data_types, - met_tool=met_tool) + actual_indices = ( + config_metplus._find_var_name_indices(config, + data_types=data_types, + met_tool=met_tool) + ) - assert len(var_name_indices) == len(expected_indices) - for actual_index in var_name_indices: + assert len(actual_indices) == len(expected_indices) + for actual_index in actual_indices: assert actual_index in expected_indices @@ -347,13 +349,13 @@ def test_parse_var_list_obs(metplus_config, data_type, list_created): # list will be created if requesting just OBS, but it should not be created if # nothing was requested because FCST values are missing if list_created: - assert(var_list[0]['obs_name'] == "NAME1" and \ - var_list[1]['obs_name'] == "NAME1" and \ - var_list[2]['obs_name'] == "NAME2" and \ - var_list[3]['obs_name'] == "NAME2" and \ - var_list[0]['obs_level'] == "LEVELS11" and \ - var_list[1]['obs_level'] == "LEVELS12" and \ - var_list[2]['obs_level'] == "LEVELS21" and \ + assert(var_list[0]['obs_name'] == "NAME1" and + var_list[1]['obs_name'] == "NAME1" and + var_list[2]['obs_name'] == "NAME2" and + var_list[3]['obs_name'] == "NAME2" and + var_list[0]['obs_level'] == "LEVELS11" and + var_list[1]['obs_level'] == "LEVELS12" and + var_list[2]['obs_level'] == "LEVELS21" and var_list[3]['obs_level'] == "LEVELS22") else: assert not var_list @@ -382,15 +384,15 @@ def test_parse_var_list_both(metplus_config, data_type, list_created): var_list = config_metplus.parse_var_list(conf, time_info=None, data_type=data_type) print(f'var_list:{var_list}') for list_to_check in list_created.split(':'): - if not var_list[0][f'{list_to_check}_name'] == "NAME1" or \ - not var_list[1][f'{list_to_check}_name'] == "NAME1" or \ - not var_list[2][f'{list_to_check}_name'] == "NAME2" or \ - not var_list[3][f'{list_to_check}_name'] == "NAME2" or \ - not var_list[0][f'{list_to_check}_level'] == "LEVELS11" or \ - not var_list[1][f'{list_to_check}_level'] == "LEVELS12" or \ - not var_list[2][f'{list_to_check}_level'] == "LEVELS21" or \ - not var_list[3][f'{list_to_check}_level'] == "LEVELS22": - assert False + if (not var_list[0][f'{list_to_check}_name'] == "NAME1" or + not var_list[1][f'{list_to_check}_name'] == "NAME1" or + not var_list[2][f'{list_to_check}_name'] == "NAME2" or + not var_list[3][f'{list_to_check}_name'] == "NAME2" or + not var_list[0][f'{list_to_check}_level'] == "LEVELS11" or + not var_list[1][f'{list_to_check}_level'] == "LEVELS12" or + not var_list[2][f'{list_to_check}_level'] == "LEVELS21" or + not var_list[3][f'{list_to_check}_level'] == "LEVELS22"): + assert False # field info defined in both FCST_* and OBS_* variables @@ -412,21 +414,21 @@ def test_parse_var_list_fcst_and_obs(metplus_config): var_list = config_metplus.parse_var_list(conf) - assert(var_list[0]['fcst_name'] == "FNAME1" and \ - var_list[0]['obs_name'] == "ONAME1" and \ - var_list[1]['fcst_name'] == "FNAME1" and \ - var_list[1]['obs_name'] == "ONAME1" and \ - var_list[2]['fcst_name'] == "FNAME2" and \ - var_list[2]['obs_name'] == "ONAME2" and \ - var_list[3]['fcst_name'] == "FNAME2" and \ - var_list[3]['obs_name'] == "ONAME2" and \ - var_list[0]['fcst_level'] == "FLEVELS11" and \ - var_list[0]['obs_level'] == "OLEVELS11" and \ - var_list[1]['fcst_level'] == "FLEVELS12" and \ - var_list[1]['obs_level'] == "OLEVELS12" and \ - var_list[2]['fcst_level'] == "FLEVELS21" and \ - var_list[2]['obs_level'] == "OLEVELS21" and \ - var_list[3]['fcst_level'] == "FLEVELS22" and \ + assert(var_list[0]['fcst_name'] == "FNAME1" and + var_list[0]['obs_name'] == "ONAME1" and + var_list[1]['fcst_name'] == "FNAME1" and + var_list[1]['obs_name'] == "ONAME1" and + var_list[2]['fcst_name'] == "FNAME2" and + var_list[2]['obs_name'] == "ONAME2" and + var_list[3]['fcst_name'] == "FNAME2" and + var_list[3]['obs_name'] == "ONAME2" and + var_list[0]['fcst_level'] == "FLEVELS11" and + var_list[0]['obs_level'] == "OLEVELS11" and + var_list[1]['fcst_level'] == "FLEVELS12" and + var_list[1]['obs_level'] == "OLEVELS12" and + var_list[2]['fcst_level'] == "FLEVELS21" and + var_list[2]['obs_level'] == "OLEVELS21" and + var_list[3]['fcst_level'] == "FLEVELS22" and var_list[3]['obs_level'] == "OLEVELS22") @@ -520,22 +522,25 @@ def test_parse_var_list_fcst_only_options(metplus_config, data_type, list_len): @pytest.mark.parametrize( 'met_tool, indices', [ - (None, {'1': ['FCST']}), - ('GRID_STAT', {'2': ['FCST']}), - ('ENSEMBLE_STAT', {}), + (None, [1]), + ('GRID_STAT', [2]), + ('ENSEMBLE_STAT', []), ] ) @pytest.mark.util def test_find_var_indices_wrapper_specific(metplus_config, met_tool, indices): - conf = metplus_config + config = metplus_config data_type = 'FCST' - conf.set('config', f'{data_type}_VAR1_NAME', "NAME1") - conf.set('config', f'{data_type}_GRID_STAT_VAR2_NAME', "GSNAME2") + config.set('config', f'{data_type}_VAR1_NAME', "NAME1") + config.set('config', f'{data_type}_GRID_STAT_VAR2_NAME', "GSNAME2") - var_name_indices = config_metplus._find_var_name_indices(conf,data_types=[data_type], - met_tool=met_tool) + actual_indices = ( + config_metplus._find_var_name_indices(config, + data_types=[data_type], + met_tool=met_tool) + ) - assert var_name_indices == indices + assert actual_indices == indices # ensure that the field configuration used for @@ -572,28 +577,28 @@ def test_parse_var_list_ensemble(metplus_config): 'ens_phist_bin_size = 0.05;')) time_info = {} - expected_ens_list = [{'index': '1', + expected_ens_list = [{'index': 1, 'ens_name': 'APCP', 'ens_level': 'A24', 'ens_thresh': ['>0.0', '>=10.0']}, - {'index': '2', + {'index': 2, 'ens_name': 'REFC', 'ens_level': 'L0', 'ens_thresh': ['>35.0']}, - {'index': '3', + {'index': 3, 'ens_name': 'UGRD', 'ens_level': 'Z10', 'ens_thresh': ['>=5.0']}, - {'index': '4', + {'index': 4, 'ens_name': 'VGRD', 'ens_level': 'Z10', 'ens_thresh': ['>=5.0']}, - {'index': '5', + {'index': 5, 'ens_name': 'WIND', 'ens_level': 'Z10', 'ens_thresh': ['>=5.0']}, ] - expected_var_list = [{'index': '1', + expected_var_list = [{'index': 1, 'fcst_name': 'APCP', 'fcst_level': 'A24', 'fcst_thresh': ['>0.01', '>=10.0'], @@ -646,7 +651,7 @@ def test_parse_var_list_series_by(metplus_config): config.set('config', 'BOTH_SERIES_ANALYSIS_VAR2_LEVELS', 'P700') time_info = {} - expected_et_list = [{'index': '1', + expected_et_list = [{'index': 1, 'fcst_name': 'RH', 'fcst_level': 'P850', 'fcst_output_name': 'RH_850mb', @@ -654,7 +659,7 @@ def test_parse_var_list_series_by(metplus_config): 'obs_level': 'P850', 'obs_output_name': 'RH_850mb', }, - {'index': '1', + {'index': 1, 'fcst_name': 'RH', 'fcst_level': 'P700', 'fcst_output_name': 'RH_700mb', @@ -663,13 +668,13 @@ def test_parse_var_list_series_by(metplus_config): 'obs_output_name': 'RH_700mb', }, ] - expected_sa_list = [{'index': '1', + expected_sa_list = [{'index': 1, 'fcst_name': 'RH_850mb', 'fcst_level': 'P850', 'obs_name': 'RH_850mb', 'obs_level': 'P850', }, - {'index': '2', + {'index': 2, 'fcst_name': 'RH_700mb', 'fcst_level': 'P700', 'obs_name': 'RH_700mb', @@ -913,3 +918,19 @@ def test_format_var_items_options_semicolon(config_value, var_items = config_metplus._format_var_items(field_configs, time_info) result = var_items.get('extra') assert result == expected_result + + +@pytest.mark.util +def test_parse_var_list_double_digit(metplus_config): + """!This test ensures that parse_var_list returns field info in + numeric order (1,2,...,9,10,11) instead of alphabetical (1,10,11,2,3,etc) + """ + config = metplus_config + for n in range(1, 12, 1): + config.set('config', f'FCST_VAR{n}_NAME', f'fcst_name{n}') + config.set('config', f'OBS_VAR{n}_NAME', f'obs_name{n}') + + var_list = config_metplus.parse_var_list(config) + for n, var_item in enumerate(var_list, start=1): + assert var_item['fcst_name'] == f'fcst_name{n}' + assert var_item['obs_name'] == f'obs_name{n}' diff --git a/internal/tests/pytests/util/config_util/test_config_util.py b/internal/tests/pytests/util/config_util/test_config_util.py index aee78545f2..7409f62b4e 100644 --- a/internal/tests/pytests/util/config_util/test_config_util.py +++ b/internal/tests/pytests/util/config_util/test_config_util.py @@ -115,13 +115,13 @@ def test_get_process_list_instances(metplus_config, input_list, expected_list): ({'init': datetime(2019, 2, 1, 6), 'lead': 7200, }, [ - {'index': '1', + {'index': 1, 'fcst_name': 'FNAME_2019', 'fcst_level': 'Z06', 'obs_name': 'ONAME_2019', 'obs_level': 'L06', }, - {'index': '1', + {'index': 1, 'fcst_name': 'FNAME_2019', 'fcst_level': 'Z08', 'obs_name': 'ONAME_2019', @@ -131,13 +131,13 @@ def test_get_process_list_instances(metplus_config, input_list, expected_list): ({'init': datetime(2021, 4, 13, 9), 'lead': 10800, }, [ - {'index': '1', + {'index': 1, 'fcst_name': 'FNAME_2021', 'fcst_level': 'Z09', 'obs_name': 'ONAME_2021', 'obs_level': 'L09', }, - {'index': '1', + {'index': 1, 'fcst_name': 'FNAME_2021', 'fcst_level': 'Z12', 'obs_name': 'ONAME_2021', diff --git a/internal/tests/pytests/wrappers/command_builder/test_command_builder.py b/internal/tests/pytests/wrappers/command_builder/test_command_builder.py index 49526476c7..6fd2d66576 100644 --- a/internal/tests/pytests/wrappers/command_builder/test_command_builder.py +++ b/internal/tests/pytests/wrappers/command_builder/test_command_builder.py @@ -855,3 +855,60 @@ def test_get_field_info_extra(metplus_config, extra, expected_value): add_curly_braces=False )[0] assert actual_value == expected_value + + +@pytest.mark.parametrize( + 'exists, skip, is_dir, use_prefix, run', [ + (True, True, False, True, False), + (True, False, False, True, True), + (False, True, False, True, True), + (False, False, False, True, True), + (True, True, True, True, False), + (True, False, True, True, True), + (False, True, True, True, True), + (False, False, True, True, True), + (True, True, False, False, False), + (True, False, False, False, True), + (False, True, False, False, True), + (False, False, False, False, True), + (True, True, True, False, False), + (True, False, True, False, True), + (False, True, True, False, True), + (False, False, True, False, True), + ] +) +@pytest.mark.wrapper +def test_find_and_check_output_file_skip(metplus_config, exists, skip, is_dir, + use_prefix, run): + app_name = 'command_builder' + config = metplus_config + if use_prefix: + config.set('config', f'{app_name.upper()}_OUTPUT_PREFIX', 'prefix') + wrapper = CommandBuilder(config) + wrapper.app_name = app_name + prefix = f'{app_name}_prefix' if use_prefix else app_name + exist_file = f'{prefix}_120000L_20190201_000000V.stat' + non_exist_file = f'{prefix}_240000L_20190201_000000V.stat' + + # create fake file to test + create_fullpath = os.path.join(config.getdir('OUTPUT_BASE'), exist_file) + open(create_fullpath, 'a').close() + + # set time_info, output template/dir, skip if output exists flag + task_info = {'valid': datetime.datetime(2019, 2, 1, 0)} + if is_dir: + task_info['lead_hours'] = 12 if exists else 24 + + time_info = ti_calculate(task_info) + wrapper.c_dict['OUTPUT_DIR'] = wrapper.config.getdir('OUTPUT_BASE') + + wrapper.c_dict['SKIP_IF_OUTPUT_EXISTS'] = skip + if is_dir: + wrapper.c_dict['OUTPUT_TEMPLATE'] = '' + else: + wrapper.c_dict['OUTPUT_TEMPLATE'] = exist_file if exists else non_exist_file + + result = wrapper.find_and_check_output_file(time_info, is_directory=is_dir) + + # cast result to bool because None isn't equal to False + assert bool(result) == run diff --git a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py index 02e35d22d4..5e612e93ce 100644 --- a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py @@ -23,41 +23,6 @@ def pb2nc_wrapper(metplus_config): return PB2NCWrapper(config) -@pytest.mark.parametrize( - 'exists, skip, run', [ - (True, True, False), - (True, False, True), - (False, True, True), - (False, False, True), - ] -) -@pytest.mark.wrapper -def test_find_and_check_output_file_skip(metplus_config, exists, skip, run): - pb = pb2nc_wrapper(metplus_config) - exist_file = 'wackyfilenametocreate' - non_exist_file = 'wackyfilethatdoesntexist' - - # create fake file to test - create_fullpath = os.path.join(pb.config.getdir('OUTPUT_BASE'), exist_file) - open(create_fullpath, 'a').close() - - # set time_info, output template/dir, skip if output exists flag - time_info = { 'valid' : datetime.datetime(2019, 2, 1, 0) } - pb.c_dict['OUTPUT_DIR'] = pb.config.getdir('OUTPUT_BASE') - - pb.c_dict['SKIP_IF_OUTPUT_EXISTS'] = skip - - if exists: - pb.c_dict['OUTPUT_TEMPLATE'] = exist_file - else: - pb.c_dict['OUTPUT_TEMPLATE'] = non_exist_file - - result = pb.find_and_check_output_file(time_info) - - # cast result to bool because None isn't equal to False - assert bool(result) == run - - # --------------------- # test_get_command # test that command is generated correctly diff --git a/internal/tests/use_cases/all_use_cases.txt b/internal/tests/use_cases/all_use_cases.txt index ca8f1cbc1c..eef745abc6 100644 --- a/internal/tests/use_cases/all_use_cases.txt +++ b/internal/tests/use_cases/all_use_cases.txt @@ -153,6 +153,7 @@ Category: short_range 10::UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview::model_applications/short_range/UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.conf:: metplotpy_env 11::UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile::model_applications/short_range/UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile.conf:: metplotpy_env 12::UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection::model_applications/short_range/UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection.conf:: metplotpy_env +13::MODEMultivar_fcstHRRR_obsMRMS_HRRRanl::model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf Category: space_weather diff --git a/metplus/util/config_metplus.py b/metplus/util/config_metplus.py index e3a2dd7ead..6d0fa4295b 100644 --- a/metplus/util/config_metplus.py +++ b/metplus/util/config_metplus.py @@ -942,9 +942,9 @@ def parse_var_list(config, time_info=None, data_type=None, met_tool=None, # get indices of VAR items for data type and/or met tool indices = [] if met_tool: - indices = _find_var_name_indices(config, data_types, met_tool).keys() + indices = _find_var_name_indices(config, data_types, met_tool) if not indices: - indices = _find_var_name_indices(config, data_types).keys() + indices = _find_var_name_indices(config, data_types) # get config name prefixes for each data type to find dt_search_prefixes = {} @@ -979,7 +979,7 @@ def parse_var_list(config, time_info=None, data_type=None, met_tool=None, # check if number of levels for each field type matches n_levels = len(field_info_list[0]['levels']) if len(data_types) > 1: - if (n_levels != len(field_info_list[1]['levels'])): + if n_levels != len(field_info_list[1]['levels']): continue # if requested, put all field levels in a single item @@ -1064,7 +1064,22 @@ def parse_var_list(config, time_info=None, data_type=None, met_tool=None, ''' return sorted(var_list, key=lambda x: x['index']) + def _find_var_name_indices(config, data_types, met_tool=None): + """!Get list of indices used in _VAR_ config variables. Data type + determines prefix of variable name to find. If FCST or OBS is included + in data type list, then BOTH keyword is also searched. If specified, + wrapper-specific variables are searched, e.g. FCST_GRID_STAT_VAR_*. + Variables that end with NAME, INPUT_FIELD_NAME, or FIELD_NAME are used to + gather indices. + + @param config METplusConfig object to read + @param data_types list of prefixes of config variables that describe the + type of data e.g. FCST or OBS. + @param met_tool (optional) name of wrapper to search for wrapper-specific + variables, e.g. *_GRID_STAT_VAR_*. + @returns list of integers for all matching config variables + """ data_type_regex = f"{'|'.join(data_types)}" # if data_types includes FCST or OBS, also search for BOTH @@ -1080,10 +1095,11 @@ def _find_var_name_indices(config, data_types, met_tool=None): regex_string += r"_VAR(\d+)_(NAME|INPUT_FIELD_NAME|FIELD_NAME)" # find all _VAR_NAME keys in the conf files - return find_indices_in_config_section(regex_string, - config, - index_index=2, - id_index=1) + indices = find_indices_in_config_section(regex_string, + config, + index_index=2, + id_index=1).keys() + return [int(index) for index in indices] def _format_var_items(field_configs, time_info=None): @@ -1210,7 +1226,7 @@ def get_field_config_variables(config, index, search_prefixes): in RegridDataPlane wrapper. @param config METplusConfig object to search - @param index of field (VAR) to find + @param index integer of field (VAR) to find @param search_prefixes list of valid prefixes to search for variables in the config, i.e. FCST_VAR1_ or OBS_GRID_STAT_VAR2_ @returns dictionary containing a config variable name to be used for diff --git a/metplus/util/config_util.py b/metplus/util/config_util.py index fa835bb6cc..5baea3623c 100644 --- a/metplus/util/config_util.py +++ b/metplus/util/config_util.py @@ -163,8 +163,8 @@ def write_all_commands(all_commands, config): @returns False if no commands were provided, True otherwise """ if not all_commands: - config.logger.error("No commands were run. " - "Skip writing all_commands file") + config.logger.info("No commands were run. " + "Skip writing all_commands file") return False log_timestamp = config.getstr('config', 'LOG_TIMESTAMP') @@ -212,6 +212,8 @@ def _sub_var_info(var_info, time_info): out_value.append(do_string_sub(item, skip_missing_tags=True, **time_info)) + elif isinstance(value, int): + out_value = value else: out_value = do_string_sub(value, skip_missing_tags=True, diff --git a/metplus/util/run_util.py b/metplus/util/run_util.py index e201392f42..9a98f91098 100644 --- a/metplus/util/run_util.py +++ b/metplus/util/run_util.py @@ -147,9 +147,7 @@ def run_metplus(config): # if process list contains any wrapper that should run commands if any([item[0] not in NO_COMMAND_WRAPPERS for item in process_list]): # write out all commands and environment variables to file - if not write_all_commands(all_commands, config): - # report an error if no commands were generated - total_errors += 1 + write_all_commands(all_commands, config) # compute total number of errors that occurred and output results for process in processes: diff --git a/metplus/util/string_manip.py b/metplus/util/string_manip.py index 3ddc53e02d..da72ca9ca0 100644 --- a/metplus/util/string_manip.py +++ b/metplus/util/string_manip.py @@ -534,11 +534,9 @@ def find_indices_in_config_section(regex, config, sec='config', @param index_index 1 based number that is the regex match index for the index number (default is 1) @param id_index 1 based number that is the regex match index for the - identifier. Defaults to None which does not extract an indentifier - - number and the first match is used as an identifier + identifier. Defaults to None which does not extract an identifier @returns dictionary where keys are the index number and the value is a - list of identifiers (if noID=True) or a list containing None + list of identifiers (if id_index=None) or a list containing None """ # regex expression must have 2 () items and the 2nd item must be the index all_conf = config.keys(sec) diff --git a/metplus/wrappers/command_builder.py b/metplus/wrappers/command_builder.py index 4a9ba4131d..3595bc6b05 100755 --- a/metplus/wrappers/command_builder.py +++ b/metplus/wrappers/command_builder.py @@ -21,7 +21,7 @@ from ..util.constants import PYTHON_EMBEDDING_TYPES, COMPRESSION_EXTENSIONS from ..util import getlist, preprocess_file, loop_over_times_and_call from ..util import do_string_sub, ti_calculate, get_seconds_from_string -from ..util import get_time_from_file, shift_time_seconds +from ..util import get_time_from_file, shift_time_seconds, seconds_to_met_time from ..util import replace_config_from_section from ..util import METConfig from ..util import MISSING_DATA_VALUE @@ -917,7 +917,7 @@ def find_and_check_output_file(self, time_info=None, template @param is_directory If True, check in output directory for any files that match the pattern - {app_name}_{output_prefix}*YYYYMMDD_HHMMSSV* + {app_name}_{output_prefix}_HHMMSSL_YYYYMMDD_HHMMSSV* @param output_path_template optional filename template to use If None, build output path template from c_dict's OUTPUT_DIR and OUTPUT_TEMPLATE. Default is None @@ -955,13 +955,18 @@ def find_and_check_output_file(self, time_info=None, # get directory that the output file will exist if is_directory: parent_dir = output_path - if time_info and time_info['valid'] != '*': - valid_format = time_info['valid'].strftime('%Y%m%d_%H%M%S') - else: - valid_format = '' + valid = '*' + lead = '*' + if time_info: + if time_info['valid'] != '*': + valid = time_info['valid'].strftime('%Y%m%d_%H%M%S') + if time_info['lead'] != '*': + lead = seconds_to_met_time(time_info['lead_seconds'], + force_hms=True) prefix = self.get_output_prefix(time_info, set_env_vars=False) - search_string = f"{self.app_name}_{prefix}*{valid_format}V*" + prefix = f'{self.app_name}_{prefix}' if prefix else self.app_name + search_string = f'{prefix}_{lead}L_{valid}V*' search_path = os.path.join(output_path, search_string) if skip_if_output_exists: diff --git a/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.conf b/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.conf index 4bad3cef08..8fb960e2e0 100644 --- a/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.conf +++ b/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.conf @@ -13,6 +13,9 @@ PROCESS_LIST = UserScript(create_cbl_filelist), UserScript(script_blocking), StatAnalysis(sanal_ibls), StatAnalysis(sanal_blocks) +# use this process list if pre-processing steps are needed +# PROCESS_LIST = RegridDataPlane(regrid_fcst), RegridDataPlane(regrid_obs), PcpCombine(daily_mean_fcst), PcpCombine(daily_mean_obs), PcpCombine(running_mean_obs), PcpCombine(anomaly_obs), UserScript(create_cbl_filelist), UserScript(script_blocking) + ### # Time Info diff --git a/parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf b/parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf new file mode 100644 index 0000000000..b8361e2f90 --- /dev/null +++ b/parm/use_cases/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.conf @@ -0,0 +1,134 @@ +[config] + +# Documentation for this use-case can be found at: +# https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl.html + +# Processes to run +PROCESS_LIST = MODE(mv),GenVxMask(fcst_super),GenVxMask(obs_super),MODE(super) + +# Time Info +LOOP_ORDER = times +LOOP_BY = INIT + +INIT_TIME_FMT = %Y%m%d%H +INIT_BEG = 2021020100 +INIT_END = 2021020100 + +LEAD_SEQ = 21 + +MODEL = HRRR +OBTYPE = ANALYSIS + +################################## +# Multivariate MODE Configurations +################################## +# Run MODE to output super objects +[mv] +MODE_MULTIVAR_LOGIC = #1 && #2 && #3 + +FCST_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl +FCST_MODE_INPUT_TEMPLATE = hrrr.t{init?fmt=%H}z.wrfprsf{lead?fmt=%H}.sub.grib2,hrrr.t{init?fmt=%H}z.wrfprsf{lead?fmt=%H}.sub.grib2,hrrr.t{init?fmt=%H}z.wrfprsf{lead?fmt=%H}.sub.grib2 + +OBS_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl +OBS_MODE_INPUT_TEMPLATE = PrecipFlag_00.00_{valid?fmt=%Y%m%d}-{valid?fmt=%2H}0000.sub.grib2,hrrr.t{valid?fmt=%H}z.wrfprsf00.sub.grib2,hrrr.t{valid?fmt=%H}z.wrfprsf00.sub.grib2 + +MODE_OUTPUT_DIR = {OUTPUT_BASE}/mode +MODE_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/f{lead?fmt=%2H} + +FCST_VAR1_NAME = CSNOW +FCST_VAR1_LEVELS = L0 +FCST_VAR1_OPTIONS = conv_radius = 0; conv_thresh = ==1 + +OBS_VAR1_NAME = PrecipFlag +OBS_VAR1_LEVELS = L0 +OBS_VAR1_OPTIONS = conv_radius = 0; conv_thresh = ==3 + +FCST_VAR2_NAME = VIS +FCST_VAR2_LEVELS = L0 +FCST_VAR2_OPTIONS = conv_radius = 5; conv_thresh = <=804.672; merge_thresh = <=1207.008; merge_flag = THRESH + +OBS_VAR2_NAME = VIS +OBS_VAR2_LEVELS = L0 +OBS_VAR2_OPTIONS = conv_radius = 5; conv_thresh = <=804.672; merge_thresh = <=1207.008; merge_flag = THRESH + +FCST_VAR3_NAME = WIND +FCST_VAR3_LEVELS = Z10 +FCST_VAR3_OPTIONS = conv_radius = 5; conv_thresh = >=8.9408; merge_thresh = >=6.7056; merge_flag = THRESH + +OBS_VAR3_NAME = WIND +OBS_VAR3_LEVELS = Z10 +OBS_VAR3_OPTIONS = conv_radius = 5; conv_thresh = >=8.9408; merge_thresh = >=6.7056; merge_flag = THRESH + +MODE_FCST_FILTER_ATTR_NAME = AREA +MODE_FCST_FILTER_ATTR_THRESH = >=25 +MODE_OBS_FILTER_ATTR_NAME = AREA +MODE_OBS_FILTER_ATTR_THRESH = >=25 + +MODE_MATCH_FLAG = MERGE_BOTH + +MODE_REGRID_TO_GRID = FCST +MODE_REGRID_METHOD = NEAREST +MODE_REGRID_WIDTH = 1 +MODE_REGRID_VLD_THRESH = 0.5 + +########################## +# GenVxMask configurations +########################## +# Mask fcst field with the fcst super object field +[fcst_super] +GEN_VX_MASK_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl +GEN_VX_MASK_INPUT_TEMPLATE = hrrr.t{init?fmt=%H}z.wrfprsf{lead?fmt=%H}.sub.grib2 +GEN_VX_MASK_INPUT_MASK_DIR = {OUTPUT_BASE}/mode +GEN_VX_MASK_INPUT_MASK_TEMPLATE = {init?fmt=%Y%m%d%H}/f{lead?fmt=%2H}/f_super.nc +GEN_VX_MASK_OPTIONS = -type data -input_field 'name="WIND";level="Z10";' -mask_field 'name="super";level="L0";' -thresh 'eq0' -value -9999 -name 'WIND_super' +GEN_VX_MASK_OUTPUT_DIR = {OUTPUT_BASE}/gen_vx_mask +GEN_VX_MASK_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/fcst_wind_super_{init?fmt=%Y%m%d%H}_f{lead?fmt=%2H}.nc + +# Mask obs field with the obs super objects +[obs_super] +GEN_VX_MASK_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/MODEMultivar_fcstHRRR_obsMRMS_HRRRanl +GEN_VX_MASK_INPUT_TEMPLATE = hrrr.t{valid?fmt=%H}z.wrfprsf00.sub.grib2 +GEN_VX_MASK_INPUT_MASK_DIR = {OUTPUT_BASE}/mode +GEN_VX_MASK_INPUT_MASK_TEMPLATE = {init?fmt=%Y%m%d%H}/f{lead?fmt=%2H}/o_super.nc +GEN_VX_MASK_OPTIONS = -type data -input_field 'name="WIND";level="Z10";' -mask_field 'name="super";level="L0";' -thresh 'eq0' -value -9999 -name 'WIND_super' +GEN_VX_MASK_OUTPUT_DIR = {OUTPUT_BASE}/gen_vx_mask +GEN_VX_MASK_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/obs_wind_super_{valid?fmt=%Y%m%d%H}.nc + +####################### +# MODE on precip supers +####################### +[super] +FCST_VAR1_NAME = WIND_super +FCST_VAR1_LEVELS = Z10 + +OBS_VAR1_NAME = WIND_super +OBS_VAR1_LEVELS = Z10 + +MODE_CONV_RADIUS = 0 +MODE_CONV_THRESH = ne-9999 +MODE_MERGE_FLAG = NONE + +MODE_OUTPUT_PREFIX = {MODEL}_vs_{OBTYPE}_{CURRENT_OBS_NAME}_{CURRENT_OBS_LEVEL} + +MODE_GRID_RES = 3 + +MODE_NC_PAIRS_FLAG_LATLON = TRUE +MODE_NC_PAIRS_FLAG_RAW = TRUE +MODE_NC_PAIRS_FLAG_OBJECT_RAW = TRUE +MODE_NC_PAIRS_FLAG_OBJECT_ID = TRUE +MODE_NC_PAIRS_FLAG_CLUSTER_ID = TRUE +MODE_NC_PAIRS_FLAG_POLYLINES = TRUE + +MODE_QUILT = True + +MODE_PS_PLOT_FLAG = TRUE +MODE_CT_STATS_FLAG = TRUE + +FCST_MODE_INPUT_DIR = {OUTPUT_BASE}/gen_vx_mask +FCST_MODE_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/fcst_wind_super_{init?fmt=%Y%m%d%H}_f{lead?fmt=%2H}.nc + +OBS_MODE_INPUT_DIR = {OUTPUT_BASE}/gen_vx_mask +OBS_MODE_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/obs_wind_super_{valid?fmt=%Y%m%d%H}.nc + +MODE_OUTPUT_DIR = {OUTPUT_BASE}/mode +MODE_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/f{lead?fmt=%2H}