Skip to content

Commit

Permalink
Feature #1514 MADIS2NC wrapper (#2591)
Browse files Browse the repository at this point in the history
* Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported)

* remove execute permissions from image files

* Per #1514, add image for basic use case

* removed large image files that are no longer being used in documentation

* add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop

* report error if output template is not defined

* update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers

* fix warnings in documentation

* fix formatting issues

* Per #1514, add new basic use case to automated test suite

* add step to comment out version number in wrapped MET config file

* turn off use case to prepare for PR

* added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data

* removed commented code

* properly substitute template tags in all command line arguments

* properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg
  • Loading branch information
georgemccabe authored May 17, 2024
1 parent a3ce98f commit b05c082
Show file tree
Hide file tree
Showing 48 changed files with 1,375 additions and 536 deletions.
5 changes: 5 additions & 0 deletions .github/parm/use_case_groups.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@
"index_list": "30-58",
"run": false
},
{
"category": "met_tool_wrapper",
"index_list": "64",
"run": false
},
{
"category": "air_quality_and_comp",
"index_list": "0",
Expand Down
101 changes: 77 additions & 24 deletions docs/Contributors_Guide/basic_components.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,15 @@ WRAPPER_ENV_VAR_KEYS
This class variable lists all of the environment variables that are set by
the wrapper. These variables are typically referenced in the wrapped MET
config file for the tool and are named with a *METPLUS\_* prefix.
All of the variables that are referenced in the wrapped MET config file must
be listed here so that they will always be set to prevent an error when MET
reads the config file. An empty string will be set if they are not set to
another value by the wrapper.
An empty string will be set if they are not set to another value by the wrapper.
Almost all of the variables that are referenced in the wrapped MET config file
must be listed here so that they will always be set to prevent an error when MET
reads the config file. The exceptions are **METPLUS_MET_CONFIG_OVERRIDES** and
**METPLUS_TIME_OFFSET_WARNING** which are automatically set and included in the
CommandBuilder initialization step.

See :ref:`bc_add_met_config` for more information on how to determine the
values to add to this list.

DEPRECATED_WRAPPER_ENV_VAR_KEYS
--------------------------------
Expand Down Expand Up @@ -217,13 +222,78 @@ it is not necessary to set *self.isOK = False* if this function is called.
if something_else_goes_wrong:
self.isOK = False
.. _bc_find_input_files:

find_input_files function
=========================

The **find_input_files** class function should be implemented to handle the
reading of input files that were found for a given run. This function should
add files to the **self.infiles** instance variable. It should return False
if any required input files were not able to be set. It should return True
if everything was read properly.

.. _bc_set_command_line_arguments:

set_command_line_arguments function
-----------------------------------

The **set_command_line_arguments** class function should be implemented to
set any arguments to pass to the command. This function should add to the
**self.args** instance variable for each command line argument.
The *time_info* dictionary containing values to substitute into filename
template tags is passed to this function. Call the **do_string_sub** function
to substitute any values. Note: Eventually the wrappers will be refactored
to handle filename template tag substitution uniformly so substitution in
this function will no longer be necessary.

.. _bc_set_environment_variables:

set_environment_variables function
==================================

The setting of environment variables to be read into the wrapped MET config
file is now handled automatically if using the *add_met_config* functions.
See :ref:`bc_add_met_config`.

However, if additional environment variables need to be set for a specific
wrapper, the **set_environment_variables** function can be overridden.
**If it is overridden, make sure to call the parent class's implementation of
the function!** Include the following in the function::

super().set_environment_variables(time_info)

Uses add_env_var function (CommandBuilder) to set any shell environment
variables that MET or other METplus wrappers
need to be set.

.. _bc_get_command:

get_command function
====================

**get_command** assembles the command that will be run.

Some wrappers will need to define the **get_command** function to format the
command and its arguments. This function reads instance variables and returns
a string of the command to run. Note: Eventually the wrappers will be refactored
to handle this uniformly, so it will not be necessary to override this function
for each wrapper.

.. _bc_run_at_time_once:

run_at_time_once function
=========================

**run_at_time_once** runs a process for one specific time. The time depends
**run_at_time_once** runs a process for one specific time.

Previously a **run_at_time_once** function needed to be implemented for each
wrapper. Now this logic is handled in the **RuntimeFreqWrapper**.
It is typically not necessary to override this function in a new wrapper
unless the wrapper requires special logic that isn't covered in the generic
version of the function in *RuntimeFreqWrapper*.

The time depends
on the value of {APP_NAME}_RUNTIME_FREQ. Most wrappers run once per each
init or valid and forecast lead time. This function is often defined in each
wrapper to handle command setup specific to the wrapper. There is a generic
Expand Down Expand Up @@ -270,25 +340,6 @@ If a wrapper is not inheriting from RuntimeFreqWrapper or one of its child
classes, then the **run_all_times** function can be implemented in the wrapper.
This function is called when the wrapper is called.

get_command function
====================

**get_command** assembles the command that will be run.
It is defined in CommandBuilder but is overridden in most wrappers because
the command line arguments differ for each MET tool.

set_environment_variables function
==================================

Uses add_env_var function (CommandBuilder) to set any shell environment
variables that MET or other METplus wrappers
need to be set. This allows a wrapper to pass information into a MET
configuration file. The MET config file refers to the environment variables.
This is currently only set in wrappers that use MET config files, but the
other wrappers will also need to set environment variables
that are needed to be set in the environment when running, such as
MET_TMP_DIR and MET_PYTHON_EXE.

find_data/find_model/find_obs functions (in CommandBuilder)
===========================================================

Expand Down Expand Up @@ -362,6 +413,8 @@ lead, lead_seconds, lead_minutes, lead_hours, offset, offset_hours, da_init
pcp_combine uses a variety of time_util functions like ti_calculate and
ti_get_lead_string

.. _bc_add_met_config:

Adding Support for MET Configuration Variables
==============================================

Expand Down
Loading

0 comments on commit b05c082

Please sign in to comment.