diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json index ea51000c83..2b5ffbb2c3 100644 --- a/.github/parm/use_case_groups.json +++ b/.github/parm/use_case_groups.json @@ -9,6 +9,11 @@ "index_list": "30-58", "run": false }, + { + "category": "met_tool_wrapper", + "index_list": "64", + "run": false + }, { "category": "air_quality_and_comp", "index_list": "0", diff --git a/docs/Contributors_Guide/basic_components.rst b/docs/Contributors_Guide/basic_components.rst index 0b24209d85..d73dc7c98d 100644 --- a/docs/Contributors_Guide/basic_components.rst +++ b/docs/Contributors_Guide/basic_components.rst @@ -86,10 +86,15 @@ WRAPPER_ENV_VAR_KEYS This class variable lists all of the environment variables that are set by the wrapper. These variables are typically referenced in the wrapped MET config file for the tool and are named with a *METPLUS\_* prefix. -All of the variables that are referenced in the wrapped MET config file must -be listed here so that they will always be set to prevent an error when MET -reads the config file. An empty string will be set if they are not set to -another value by the wrapper. +An empty string will be set if they are not set to another value by the wrapper. +Almost all of the variables that are referenced in the wrapped MET config file +must be listed here so that they will always be set to prevent an error when MET +reads the config file. The exceptions are **METPLUS_MET_CONFIG_OVERRIDES** and +**METPLUS_TIME_OFFSET_WARNING** which are automatically set and included in the +CommandBuilder initialization step. + +See :ref:`bc_add_met_config` for more information on how to determine the +values to add to this list. DEPRECATED_WRAPPER_ENV_VAR_KEYS -------------------------------- @@ -217,13 +222,78 @@ it is not necessary to set *self.isOK = False* if this function is called. if something_else_goes_wrong: self.isOK = False +.. _bc_find_input_files: + +find_input_files function +========================= + +The **find_input_files** class function should be implemented to handle the +reading of input files that were found for a given run. This function should +add files to the **self.infiles** instance variable. It should return False +if any required input files were not able to be set. It should return True +if everything was read properly. + +.. _bc_set_command_line_arguments: + +set_command_line_arguments function +----------------------------------- + +The **set_command_line_arguments** class function should be implemented to +set any arguments to pass to the command. This function should add to the +**self.args** instance variable for each command line argument. +The *time_info* dictionary containing values to substitute into filename +template tags is passed to this function. Call the **do_string_sub** function +to substitute any values. Note: Eventually the wrappers will be refactored +to handle filename template tag substitution uniformly so substitution in +this function will no longer be necessary. + +.. _bc_set_environment_variables: + +set_environment_variables function +================================== + +The setting of environment variables to be read into the wrapped MET config +file is now handled automatically if using the *add_met_config* functions. +See :ref:`bc_add_met_config`. + +However, if additional environment variables need to be set for a specific +wrapper, the **set_environment_variables** function can be overridden. +**If it is overridden, make sure to call the parent class's implementation of +the function!** Include the following in the function:: + + super().set_environment_variables(time_info) + +Uses add_env_var function (CommandBuilder) to set any shell environment +variables that MET or other METplus wrappers +need to be set. + +.. _bc_get_command: + +get_command function +==================== + +**get_command** assembles the command that will be run. + +Some wrappers will need to define the **get_command** function to format the +command and its arguments. This function reads instance variables and returns +a string of the command to run. Note: Eventually the wrappers will be refactored +to handle this uniformly, so it will not be necessary to override this function +for each wrapper. .. _bc_run_at_time_once: run_at_time_once function ========================= -**run_at_time_once** runs a process for one specific time. The time depends +**run_at_time_once** runs a process for one specific time. + +Previously a **run_at_time_once** function needed to be implemented for each +wrapper. Now this logic is handled in the **RuntimeFreqWrapper**. +It is typically not necessary to override this function in a new wrapper +unless the wrapper requires special logic that isn't covered in the generic +version of the function in *RuntimeFreqWrapper*. + +The time depends on the value of {APP_NAME}_RUNTIME_FREQ. Most wrappers run once per each init or valid and forecast lead time. This function is often defined in each wrapper to handle command setup specific to the wrapper. There is a generic @@ -270,25 +340,6 @@ If a wrapper is not inheriting from RuntimeFreqWrapper or one of its child classes, then the **run_all_times** function can be implemented in the wrapper. This function is called when the wrapper is called. -get_command function -==================== - -**get_command** assembles the command that will be run. -It is defined in CommandBuilder but is overridden in most wrappers because -the command line arguments differ for each MET tool. - -set_environment_variables function -================================== - -Uses add_env_var function (CommandBuilder) to set any shell environment -variables that MET or other METplus wrappers -need to be set. This allows a wrapper to pass information into a MET -configuration file. The MET config file refers to the environment variables. -This is currently only set in wrappers that use MET config files, but the -other wrappers will also need to set environment variables -that are needed to be set in the environment when running, such as -MET_TMP_DIR and MET_PYTHON_EXE. - find_data/find_model/find_obs functions (in CommandBuilder) =========================================================== @@ -362,6 +413,8 @@ lead, lead_seconds, lead_minutes, lead_hours, offset, offset_hours, da_init pcp_combine uses a variety of time_util functions like ti_calculate and ti_get_lead_string +.. _bc_add_met_config: + Adding Support for MET Configuration Variables ============================================== diff --git a/docs/Contributors_Guide/create_wrapper.rst b/docs/Contributors_Guide/create_wrapper.rst index 287ffa0ea7..5edd7b8350 100644 --- a/docs/Contributors_Guide/create_wrapper.rst +++ b/docs/Contributors_Guide/create_wrapper.rst @@ -11,7 +11,8 @@ File Name Create the new wrapper in the *metplus/wrappers* directory and name it to reflect the wrapper's function, e.g.: new_tool_wrapper.py is a wrapper around an application named "new_tool." -Copy the **example_wrapper.py** to start the process. +Identify an existing wrapper that is similar to the new wrapper and +copy it to start the process. Class Name ---------- @@ -57,16 +58,119 @@ MTD or accidentally write PointToGrid instead of Point2Grid:: More than one entry is rarely needed, but they will not hurt anything as long as they do not cause any conflicts. +.. _cw_wrapped_met_config: + +Wrapped MET Config File +======================= + +If the new wrapper corresponds to a MET tool that supports a configuration file, +then a wrapped version of the MET configuration file should be created. +If unsure if the MET tool supports a configuration file, check the +MET User's Guide section for the tool. Alternatively, look for a +file ending with *\_default* that corresponds to the MET tool in the +`data/config `_ +directory of the MET repository. +If this file exists, then the tool supports a configuration file. + +If the new wrapper is for a MET tool that does not support a configuration file +or the wrapper does not correspond to a MET tool at all, then this section can +be skipped. + +Copy Default Config File +------------------------ + +Copy the appropriate default configuration file found in the +`data/config `_ +directory of the MET repository into the parm/met_config directory of the +METplus repository and rename it to end with *_wrapped* instead of *_default*. +For example, if creating the GridStat wrapper, +copy MET/data/config/GridStatConfig\_**default** to +METplus/parm/met_config/GridStatConfig\_**wrapped**. + +**MAKE SURE TO COPY THE DEFAULT CONFIG FILE FROM THE DEVELOP BRANCH TO GET +THE LATEST UPDATES** + +Remove Apostrophe +----------------- + +**REMOVE THE \' CHARACTER FROM THE HEADER COMMENTS TO PREVENT ERRORS RENDERING +THE DOCUMENTATION THAT INCLUDES THE WRAPPED CONFIG FILE** + +Change this line (typically the 5th line):: + + // For additional information, please see the MET User's Guide. + +to:: + + // For additional information, please see the MET Users Guide. + +Changes for All Wrappers +------------------------ + +There are a few changes to make to the wrapped MET config file for all tools. +These changes are found at the very end of the config file. + +tmp_dir +^^^^^^^ + +Set the value of *tmp_dir* to the ${MET_TMP_DIR} environment variable. + +Change:: + + tmp_dir = "/tmp"; + +to:: + + tmp_dir = "${MET_TMP_DIR}"; + + +Remove Version +^^^^^^^^^^^^^^ + +The **version** variable should not be set, as it will conflict with newer +versions of MET. However, it can be nice to keep the variable commented-out +in the wrapped file to easily see roughly when the new wrapper was added. + +Change:: + + version = "V12.0.0"; + +to:: + + //version = "V12.0.0"; + +Common Environment Variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All wrappers automatically support setting *time_offset_warning* and +:ref:`met-config-overrides`. + +Add the following to the very end of the wrapped MET config file:: + + ${METPLUS_TIME_OFFSET_WARNING} + ${METPLUS_MET_CONFIG_OVERRIDES} + +If these variables are not added, these values will not be read by the MET tool. + + Wrapper Components ================== Open the wrapper file for editing the new class. -Naming ------- +Naming/Parent Class +------------------- Rename the class to match the wrapper's class from the above sections. -Most wrappers should be a subclass of the RuntimeFreqWrapper:: +If the new tool falls under one of the existing tool categories, +then make the tool a subclass of one of the existing classes. +This should only be done if the functions in the parent class are needed +by the new wrapper. When in doubt, use the **RuntimeFreqWrapper**. + +See :ref:`bc_class_hierarchy` for more information on existing classes to +determine which class to use as the parent class. + +Example:: class NewToolWrapper(RuntimeFreqWrapper) @@ -80,24 +184,18 @@ To create EnsembleStat wrapper from GridStat, replace **grid_stat** with **ensemble_stat** and **GridStat** with **EnsembleStat**. -Parent Class ------------- - -If the new tool falls under one of the existing tool categories, -then make the tool a subclass of one of the existing classes. -This should only be done if the functions in the parent class are needed -by the new wrapper. When in doubt, use the **RuntimeFreqWrapper**. - -See :ref:`bc_class_hierarchy` for more information on existing classes to -determine which class to use as the parent class. -Class Variables for Runtime Frequency -------------------------------------- +Class Variables +--------------- **RUNTIME_FREQ_DEFAULT** and **RUNTIME_FREQ_SUPPORTED** should be set for all -wrappers that inherit from **RuntimeFreqWrapper**. +wrappers that inherit from **RuntimeFreqWrapper** or one of its sub-classes. -See :ref:`bc_class_vars` for more information. +If the tool can read a config file, then **WRAPPER_ENV_VAR_KEYS** should be +defined to include a list of the environment variables that will be read +by the wrapped config file. + +See :ref:`bc_class_vars` for more information on how to set these variables. Init Function ------------- @@ -132,11 +230,6 @@ The function should also always return the c_dict variable:: return c_dict -File Input/Output -^^^^^^^^^^^^^^^^^ - -METplus configuration variables that end with _DIR and _TEMPLATE are used -to define the criteria to search for input files. Allow Multiple Files ^^^^^^^^^^^^^^^^^^^^ @@ -151,95 +244,166 @@ If it is set to False and a list of files are found for an input (using wildcards or a list of files in the METplus config template variable) then the wrapper will produce an error and not build the command. -Run Functions -------------- +Input Files +^^^^^^^^^^^ + +METplus configuration variables that end with **\_INPUT_DIR** and +**\_INPUT_TEMPLATE** are used to search for input files. + +The **get_input_templates** function can be used to easily set up the wrapper +to read the appropriate METplus config variables for inputs. +The first argument is the c_dict variable, which will be modified by the +function. +The 2nd argument is a dictionary that defines the inputs. The key is the name +of the input type, e.g. *FCST* or *OBS*. The value is a dictionary that must +include at least the *prefix* key which defines the prefix of the METplus +configuration variables to read, +e.g. **{prefix}_INPUT_DIR** and **{prefix}_INPUT_TEMPLATE**. + +The *required* key can be set to specify if the input must be defined in the +METplus config file or not. +If set to True, an error is reported if the *{prefix}_INPUT_TEMPLATE* variable +is not set. If the *required* key is not defined, the default value is True. + +Example 1 (single observation input):: + + self.get_input_templates(c_dict, { + 'OBS': {'prefix': 'MADIS2NC', 'required': True}, + }) + +This will read the METplus config variables **MADIS2NC_INPUT_DIR** and +**MADIS2NC_INPUT_TEMPLATE** and set the c_dict items **OBS_INPUT_DIR** and +**OBS_INPUT_TEMPLATE**. +An error will be reported if **MADIS2NC_INPUT_TEMPLATE** is not set. + +Example 2 (forecast and obs input):: + + self.get_input_templates(c_dict, { + 'FCST': {'prefix': 'FCST_GRID_STAT', 'required': True}, + 'OBS': {'prefix': 'OBS_GRID_STAT', 'required': True}, + }) + +This will read the METplus config variables **FCST_GRID_STAT_INPUT_DIR**, +**FCST_GRID_STAT_INPUT_TEMPLATE**, **OBS_GRID_STAT_INPUT_DIR**, and +**OBS_GRID_STAT_INPUT_TEMPLATE** and set the c_dict items **FCST_INPUT_DIR**, +**FCST_INPUT_TEMPLATE**, **OBS_INPUT_DIR**, and **OBS_INPUT_TEMPLATE**. +An error will be reported if **FCST_GRID_STAT_INPUT_TEMPLATE** or +**OBS_GRID_STAT_INPUT_TEMPLATE** is not set. + +Output Files +^^^^^^^^^^^^ + +METplus configuration variables that end with **\_OUTPUT_DIR** and +**\_OUTPUT_TEMPLATE** are used to write output files. + +Add the following and change **APP_NAME** to the name of the new wrapper:: + + c_dict['OUTPUT_DIR'] = self.config.getdir('APP_NAME_OUTPUT_DIR', '') + c_dict['OUTPUT_TEMPLATE'] = ( + self.config.getraw('config', 'APP_NAME_OUTPUT_TEMPLATE') + ) + +The *OUTPUT_DIR* and *OUTPUT_TEMPLATE* will be concatenated to form the path +to write output. + +Some MET tools write a single output file and some write multiple output files +into a directory. + +If the tool writes multiple output files, then the +*OUTPUT_TEMPLATE* is optional, but can be used to create sub-directories that +include information specific to a given run, like timestamps. + +If the tool writes a single output file, the *OUTPUT_TEMPLATE* is required. +In this case, add a check to report an error if the value is unset:: + + if not c_dict['OUTPUT_TEMPLATE']: + self.log_error('APP_NAME_OUTPUT_TEMPLATE must be set') -* The **run_at_time_once** function or some the functions that it calls will - need to be overridden in the wrapper. - See :ref:`bc_run_at_time_once` for more information. - -* It is recommended to divide up the logic into small functions to make - the code more readable and easier to test. - -* The function self.set_environment_variables should be called by all - wrappers even if the MET tool does not have a config file. - This function is typically called from the run_at_time_once function. - This is done to set environment variables that MET expects to be set when - running, such as MET_TMP_DIR and MET_PYTHON_EXE. - If no environment variables need to be set specific to the wrapper, then no - implementation of the function in the wrapper needs to be written. Call the - implementation of the function from CommandBuilder, which sets the - environment variables defined in the [user_env_vars] section of the - configuration file and outputs DEBUG logs for each environment variable - that has been set in the wrapper. MET_TMP_DIR is automatically set for - each wrapper. - -* Once all the necessary information has been provided to create the MET - command, call self.build(). This calls self.get_command() - to assemble the command and verify that the command wrapper generated - contains all of the required arguments. The get_command() in the wrapper - may need to be overridden if the MET application is different from - the example. - For instance, some MET tools require flags such as -f to - precede the input filename. The get_command function in the wrapper can be - overwritten to prepend the required flag to the filename in the - constructed MET command. - -* Call self.clear() at the beginning of each loop iteration that tries to - build/run a MET command to prevent inadvertently reusing/re-running - commands that were previously created. This is called in the RuntimeFreq - wrapper before each call to run_at_time_once, but an additional call may be - needed if multiple commands are built and run in this function. - -* To allow the use case to use the specific wrapper, assign the wrapper name to - PROCESS_LIST:: - - [config] - PROCESS_LIST = NewExample - -.. note:: - - Do not include the text "Wrapper" at the end of the wrapper name. - - Each value must match an existing wrapper name without the ‘Wrapper' - suffix. The PROCESS_LIST :numref:`Process_list` is located under the - [config] section header in the - use case and/or example configuration file. - -* Add a section to the Python Wrappers page of the documentation with - information about the new tool including a list of all METplus - configuration variables that can be used. - -* Add an entry for each METplus configuration variable added to the wrapper - to the METplus Configuration Glossary. Each configuration variable should - be the MET tool name in all caps i.e. GRID_STAT followed by the variable - name. MET tool names generally have underscores between words unless there - is a number in the name. Examples below:: - - GRID_STAT_PROB_THRESH - REGRID_DATA_PLANE_METHOD - POINT2GRID_QC_FLAGS - -* Create a directory named after the new wrapper to hold the use case - configuration files in the met_tool_wrapper directory that users can run - to try out the new wrapper. In the corresponding directory under - docs/use_cases, be sure to include a .py file that contains the - documentation for that use case and a README file to create a header for - the documentation page. - -This new use case/example configuration file is located in a directory structure -like the following:: - - parm/use_cases/met_tool_wrapper/NewTool/NewTool.conf - docs/use_cases/met_tool_wrapper/NewTool/NewTool.py - docs/use_cases/met_tool_wrapper/NewTool/README.rst - -Note the documentation file is in METplus/docs while the use case conf file -is in METplus/parm. - -Refer to the :ref:`basic_components_of_wrappers` section of the Contributor's +Config File +^^^^^^^^^^^ + +If the wrapper corresponds to a MET tool that supports a MET configuration file, +then add a call to the **get_config_file** function to handle the METplus +configuration settings. Pass the name of the wrapped MET config file that you +have added to *parm/met_config* to the function + +Example for MADIS2NC wrapper:: + + c_dict['CONFIG_FILE'] = self.get_config_file('Madis2NcConfig_wrapped') + +See :ref:`cw_wrapped_met_config` for more information. + +Add calls to **self.add_met_config** or **self.add_met_config_dict** functions +to handle the reading of METplus configuration variables to set wrapped MET +config file variables. See :ref:`bc_add_met_config` for examples and +instructions to use a helper script to determine what to set to add support +for setting a MET config variable through METplus. +If a MET config variable is already supported in another wrapper, refer to +the *create_c_dict* function for that wrapper, copying and modifying function +calls as appropriate. + +Command Line Arguments +^^^^^^^^^^^^^^^^^^^^^^ + +Add calls to *self.config.get* functions to read values from the METplus +config to set *c_dict* items that can be used to set command line arguments. +The METplus configuration variables should match the format +{APP_NAME}_{ARG_NAME} where {APP_NAME} is the name of the wrapper and {ARG_NAME} +is the name of the command line argument. Use the appropriate get function that +corresponds to the argument data type, e.g. *getraw* for strings and +*getint* for integers. + +Example:: + + c_dict['TYPE'] = self.config.getraw('config', 'MADIS2NC_TYPE') + +If the command line argument is required to run, then add a check to report an +error if the value is unset:: + + if not c_dict['TYPE']: + self.log_error('Must set MADIS2NC_TYPE') + +Make sure to remember to add logic to read the c_dict item and add the value +to the command to generate. This can be done in the *set_command_line_arguments* +class function. + +Implement Class Functions +------------------------- + +The following functions should be implemented in the new wrapper: + +* find_input_files +* set_command_line_arguments + +Some wrappers will also need to implement: + +* set_environment_variables +* get_command + +See the :ref:`basic_components_of_wrappers` chapter for more information on +how to define these functions. + +Basic Use Case Example +====================== + +The new wrapper should include a basic use case under the +*parm/use_cases/met_tool_wrapper* directory to demonstrate how to configure it. + +Following the instructions in :ref:`adding-use-cases` and refer to an existing +use case for a similar wrapper. + +Refer to the :ref:`basic_components_of_wrappers` chapter of the Contributor's Guide for more information on what should be added. +Unit Tests +========== + +Unit tests for each wrapper should be defined under +*internal/tests/pytests/wrappers*. +Create a new directory for the new wrapper. +Copy an existing test script for a similar wrapper and modify as needed to +match the new wrapper. + Documentation ============= diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst index 215ab39a8b..d268fdb793 100644 --- a/docs/Users_Guide/glossary.rst +++ b/docs/Users_Guide/glossary.rst @@ -11634,6 +11634,199 @@ METplus Configuration Glossary | *Used by:* PointStat + MADIS2NC_CUSTOM_LOOP_LIST + Sets custom string loop list for a specific wrapper. See :term:`CUSTOM_LOOP_LIST`. + + | *Used by:* MADIS2NC + + LOG_MADIS2NC_VERBOSITY + Overrides the log verbosity for MADIS2NC only. If not set, the verbosity level is controlled by :term:`LOG_MET_VERBOSITY`. + + | *Used by:* MADIS2NC + + MADIS2NC_CONFIG_FILE + Path to configuration file read by madis2nc. + If unset, parm/met_config/Madis2NcConfig_wrapped will be used. + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_IF_OUTPUT_EXISTS + If True, do not run MADIS2NC if output file already exists. Set to False to overwrite files. + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_GRID + Named grid or a data file defining the grid for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_POLY + A polyline file, the output of gen_vx_mask, or a gridded data file with field information for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_MASK_SID + A station ID masking file or a comma-separated list of station ID's for filtering the point observations spatially (optional). + + | *Used by:* MADIS2NC + + MADIS2NC_INPUT_DIR + Directory containing input data to MADIS2NC. This variable is optional because you can specify the full path to the input files using :term:`MADIS2NC_INPUT_TEMPLATE`. + + | *Used by:* MADIS2NC + + MADIS2NC_INPUT_TEMPLATE + Filename template of the input file used by MADIS2NC. See also :term:`MADIS2NC_INPUT_DIR`. + + | *Used by:* MADIS2NC + + MADIS2NC_OUTPUT_DIR + Directory to write output data generated by MADIS2NC. This variable is optional because you can specify the full path to the output files using :term:`MADIS2NC_OUTPUT_TEMPLATE`. + + | *Used by:* MADIS2NC + + MADIS2NC_OUTPUT_TEMPLATE + Filename template of the output file generated by MADIS2NC. See also :term:`MADIS2NC_OUTPUT_DIR`. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_FLAG + Specify the value for 'time_summary.flag' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_RAW_DATA + Specify the value for 'time_summary.raw_data' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_BEG + Specify the value for 'time_summary.beg' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_END + Specify the value for 'time_summary.end' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_STEP + Specify the value for 'time_summary.step' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_WIDTH + Specify the value for 'time_summary.width' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_GRIB_CODE + Specify the value for 'time_summary.grib_code' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_OBS_VAR + Specify the value for 'time_summary.obs_var' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_TYPE + Specify the value for 'time_summary.type' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_VLD_FREQ + Specify the value for 'time_summary.vld_freq' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_SUMMARY_VLD_THRESH + Specify the value for 'time_summary.vld_thresh' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_FILE_WINDOW_BEGIN + Used to control the lower bound of the window around the valid time to determine if an MADIS2NC input file should be used for processing. Overrides :term:`OBS_FILE_WINDOW_BEGIN`. See 'Use Windows to Find Valid Files' section for more information. + + | *Used by:* MADIS2NC + + MADIS2NC_FILE_WINDOW_END + Used to control the upper bound of the window around the valid time to determine if an MADIS2NC input file should be used for processing. Overrides :term:`OBS_FILE_WINDOW_END`. See 'Use Windows to Find Valid Files' section for more information. + + | *Used by:* MADIS2NC + + MADIS2NC_MET_CONFIG_OVERRIDES + Override any variables in the MET configuration file that are not + supported by the wrapper. This should be set to the full variable name + and value that you want to override, including the equal sign and the + ending semi-colon. The value is directly appended to the end of the + wrapped MET config file. + + Example: + MADIS2NC_MET_CONFIG_OVERRIDES = desc = "override_desc"; model = "override_model"; + + See :ref:`Overriding Unsupported MET config file settings` for more information + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_VALID_TIMES + List of valid times to skip for MADIS2NC only. + If set, values set in :term:`SKIP_VALID_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_INC_VALID_TIMES + List of valid times to include for MADIS2NC only. + If set, values set in :term:`INC_VALID_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_SKIP_INIT_TIMES + List of initialization times to skip for MADIS2NC only. + If set, values set in :term:`SKIP_INIT_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_INC_INIT_TIMES + List of initialization times to include for MADIS2NC only. + If set, values set in :term:`INC_INIT_TIMES` are ignored for MADIS2NC. + See :term:`SKIP_VALID_TIMES` for formatting information. + + | *Used by:* MADIS2NC + + MADIS2NC_TYPE + Specify the value for the '-type' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_QC_DD + Specify the value for the '-qc_dd' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_LVL_DIM + Specify the value for the '-lvl_dim' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_REC_BEG + Specify the value for the '-rec_beg' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_REC_END + Specify the value for the '-rec_end' command line argument for MADIS2NC. + + | *Used by:* MADIS2NC + + MADIS2NC_TIME_OFFSET_WARNING + Specify the value for 'time_offset_warning' in the MET configuration file for MADIS2NC. + + | *Used by:* MADIS2NC + ASCII2NC_TIME_OFFSET_WARNING Specify the value for 'time_offset_warning' in the MET configuration file for ASCII2NC. diff --git a/docs/Users_Guide/quicksearch.rst b/docs/Users_Guide/quicksearch.rst index a78f687ba0..30aaca532e 100644 --- a/docs/Users_Guide/quicksearch.rst +++ b/docs/Users_Guide/quicksearch.rst @@ -22,6 +22,7 @@ Use Cases by MET Tool: | `GridStat <../search.html?q=GridStatToolUseCase&check_keywords=yes&area=default>`_ | `GridDiag <../search.html?q=GridDiagToolUseCase&check_keywords=yes&area=default>`_ | `IODA2NC <../search.html?q=IODA2NCToolUseCase&check_keywords=yes&area=default>`_ + | `MADIS2NC <../search.html?q=MADIS2NCToolUseCase&check_keywords=yes&area=default>`_ | `MODE <../search.html?q=MODEToolUseCase&check_keywords=yes&area=default>`_ | `MTD <../search.html?q=MTDToolUseCase&check_keywords=yes&area=default>`_ | `PB2NC <../search.html?q=PB2NCToolUseCase&check_keywords=yes&area=default>`_ @@ -49,6 +50,7 @@ Use Cases by MET Tool: | **GridStat**: *GridStatToolUseCase* | **GridDiag**: *GridDiagToolUseCase* | **IODA2NC**: *IODA2NCToolUseCase* + | **MADIS2NC**: *MADIS2NCToolUseCase* | **MODE**: *MODEToolUseCase* | **MTD**: *MTDToolUseCase* | **PB2NC**: *PB2NCToolUseCase* diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index d35b8f031c..7f06973a12 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -4121,6 +4121,138 @@ ${METPLUS_MET_CONFIG_OVERRIDES} - n/a +.. _madis2nc_wrapper: + +MADIS2NC +======== + +Description +----------- + +Used to configure the MET tool madis2nc + +METplus Configuration +--------------------- + +| :term:`MADIS2NC_INPUT_DIR` +| :term:`MADIS2NC_OUTPUT_DIR` +| :term:`MADIS2NC_INPUT_TEMPLATE` +| :term:`MADIS2NC_OUTPUT_TEMPLATE` +| :term:`LOG_MADIS2NC_VERBOSITY` +| :term:`MADIS2NC_SKIP_IF_OUTPUT_EXISTS` +| :term:`MADIS2NC_CONFIG_FILE` +| :term:`MADIS2NC_FILE_WINDOW_BEGIN` +| :term:`MADIS2NC_FILE_WINDOW_END` +| :term:`MADIS2NC_TYPE` +| :term:`MADIS2NC_QC_DD` +| :term:`MADIS2NC_LVL_DIM` +| :term:`MADIS2NC_REC_BEG` +| :term:`MADIS2NC_REC_END` +| :term:`MADIS2NC_MASK_GRID` +| :term:`MADIS2NC_MASK_POLY` +| :term:`MADIS2NC_MASK_SID` +| :term:`MADIS2NC_TIME_SUMMARY_FLAG` +| :term:`MADIS2NC_TIME_SUMMARY_RAW_DATA` +| :term:`MADIS2NC_TIME_SUMMARY_BEG` +| :term:`MADIS2NC_TIME_SUMMARY_END` +| :term:`MADIS2NC_TIME_SUMMARY_STEP` +| :term:`MADIS2NC_TIME_SUMMARY_WIDTH` +| :term:`MADIS2NC_TIME_SUMMARY_GRIB_CODE` +| :term:`MADIS2NC_TIME_SUMMARY_OBS_VAR` +| :term:`MADIS2NC_TIME_SUMMARY_TYPE` +| :term:`MADIS2NC_TIME_SUMMARY_VLD_FREQ` +| :term:`MADIS2NC_TIME_SUMMARY_VLD_THRESH` +| :term:`MADIS2NC_CUSTOM_LOOP_LIST` +| :term:`MADIS2NC_MET_CONFIG_OVERRIDES` +| :term:`MADIS2NC_SKIP_VALID_TIMES` +| :term:`MADIS2NC_INC_VALID_TIMES` +| :term:`MADIS2NC_SKIP_INIT_TIMES` +| :term:`MADIS2NC_INC_INIT_TIMES` +| :term:`MADIS2NC_TIME_OFFSET_WARNING` +| + +.. _madis2nc-met-conf: + +MET Configuration +----------------- + +Below is the wrapped MET configuration file used for this wrapper. +Environment variables are used to control entries in this configuration file. +The default value for each environment variable is obtained from +(except where noted below): + +`MET_INSTALL_DIR/share/met/config/Madis2NcConfig_default `_ + +Below the file contents are descriptions of each environment variable +referenced in this file and the corresponding METplus configuration item used +to set the value of the environment variable. For detailed examples showing +how METplus sets the values of these environment variables, +see :ref:`How METplus controls MET config file settings`. + +.. dropdown:: Click to view parm/met_config/Madis2NcConfig_wrapped + + .. literalinclude:: ../../parm/met_config/Madis2NcConfig_wrapped + +Environment variables in wrapped MET config +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +${METPLUS_TIME_SUMMARY_DICT} +"""""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_TIME_SUMMARY_FLAG` + - time_summary.flag + * - :term:`MADIS2NC_TIME_SUMMARY_RAW_DATA` + - time_summary.raw_data + * - :term:`MADIS2NC_TIME_SUMMARY_BEG` + - time_summary.beg + * - :term:`MADIS2NC_TIME_SUMMARY_END` + - time_summary.end + * - :term:`MADIS2NC_TIME_SUMMARY_STEP` + - time_summary.step + * - :term:`MADIS2NC_TIME_SUMMARY_WIDTH` + - time_summary.width + * - :term:`MADIS2NC_TIME_SUMMARY_GRIB_CODE` + - time_summary.grib_code + * - :term:`MADIS2NC_TIME_SUMMARY_OBS_VAR` + - time_summary.obs_var + * - :term:`MADIS2NC_TIME_SUMMARY_TYPE` + - time_summary.type + * - :term:`MADIS2NC_TIME_SUMMARY_VLD_FREQ` + - time_summary.vld_freq + * - :term:`MADIS2NC_TIME_SUMMARY_VLD_THRESH` + - time_summary.vld_thresh + +${METPLUS_TIME_OFFSET_WARNING} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_TIME_OFFSET_WARNING` + - time_offset_warning + +${METPLUS_MET_CONFIG_OVERRIDES} +""""""""""""""""""""""""""""""" + +.. list-table:: + :widths: 5 5 + :header-rows: 1 + + * - METplus Config(s) + - MET Config File + * - :term:`MADIS2NC_MET_CONFIG_OVERRIDES` + - n/a + + .. _met_db_load_wrapper: METdbLoad diff --git a/docs/_static/Estes_Park_Sky_3_March_2017.png b/docs/_static/Estes_Park_Sky_3_March_2017.png deleted file mode 100755 index f646d06581..0000000000 Binary files a/docs/_static/Estes_Park_Sky_3_March_2017.png and /dev/null differ diff --git a/docs/_static/METplus_banner_photo.png b/docs/_static/METplus_banner_photo.png deleted file mode 100755 index a33b858c87..0000000000 Binary files a/docs/_static/METplus_banner_photo.png and /dev/null differ diff --git a/docs/_static/METplus_banner_photo_web.png b/docs/_static/METplus_banner_photo_web.png old mode 100755 new mode 100644 diff --git a/docs/_static/METplus_logo.png b/docs/_static/METplus_logo.png old mode 100755 new mode 100644 diff --git a/docs/_static/data_assimilation-StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.png b/docs/_static/data_assimilation-StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.png old mode 100755 new mode 100644 diff --git a/docs/_static/download_metplus.png b/docs/_static/download_metplus.png old mode 100755 new mode 100644 diff --git a/docs/_static/marine_and_cryosphere_GridStat_MODE_fcstIMS_obsNCEP_Sea_Ice.png b/docs/_static/marine_and_cryosphere_GridStat_MODE_fcstIMS_obsNCEP_Sea_Ice.png old mode 100755 new mode 100644 diff --git a/docs/_static/met_tool_wrapper-MADIS2NC.png b/docs/_static/met_tool_wrapper-MADIS2NC.png new file mode 100644 index 0000000000..cab2ce69cc Binary files /dev/null and b/docs/_static/met_tool_wrapper-MADIS2NC.png differ diff --git a/docs/_static/met_tool_wrapper-PlotDataPlane.png b/docs/_static/met_tool_wrapper-PlotDataPlane.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo.png b/docs/_static/metplus_repo.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo_release.png b/docs/_static/metplus_repo_release.png old mode 100755 new mode 100644 diff --git a/docs/_static/metplus_repo_releases_page.png b/docs/_static/metplus_repo_releases_page.png old mode 100755 new mode 100644 diff --git a/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py b/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py new file mode 100644 index 0000000000..709edb3fcc --- /dev/null +++ b/docs/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.py @@ -0,0 +1,105 @@ +""" +MADIS2NC: Basic Use Case +======================== + +met_tool_wrapper/MADIS2NC/MADIS2NC.conf + +""" +############################################################################## +# Scientific Objective +# -------------------- +# +# Converting file formats so point observations can be read by the MET tools. + +############################################################################## +# Datasets +# -------- +# +# | **Observations:** METAR observations in MADIS NetCDF files +# +# | **Location:** All of the input data required for this use case can be found in the met_test sample data tarball. Click here to the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases +# | This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See `Running METplus`_ section for more information. +# +# | **Data Source:** MADIS +# | + +############################################################################## +# METplus Components +# ------------------ +# +# This use case utilizes the METplus MADIS2NC wrapper to generate a command to run the MET tool madis2nc if all required files are found. + +############################################################################## +# METplus Workflow +# ---------------- +# +# MADIS2NC is the only tool called in this example. It processes the following +# run time: +# +# | **Valid:** 2012-04-09_12Z +# | + +############################################################################## +# METplus Configuration +# --------------------- +# +# METplus first loads the default configuration file found in parm/metplus_config, +# then it loads any configuration files passed to METplus via the command line. +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf + +############################################################################## +# MET Configuration +# --------------------- +# +# METplus sets environment variables based on user settings in the METplus configuration file. +# See :ref:`How METplus controls MET config file settings` for more details. +# +# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!** +# +# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to: +# :ref:`Overriding Unsupported MET config file settings` +# +# .. note:: See the :ref:`MADIS2NC MET Configuration` section of the User's Guide for more information on the environment variables used in the file below: +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/met_config/Madis2NcConfig_wrapped + +############################################################################## +# Running METplus +# --------------- +# +# Pass the path to MADIS2NC.conf as an argument to run_metplus.py:: +# +# run_metplus.py /path/to/METplus/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf +# + +############################################################################## +# Expected Output +# --------------- +# +# A successful run will output the following both to the screen and to the logfile:: +# +# INFO: METplus has successfully finished running. +# +# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated. +# Output for this use case will be found in madis2nc (relative to **OUTPUT_BASE**) +# and will contain the following file: +# +# * met_metar_2012040912_F000.nc + +############################################################################## +# Keywords +# -------- +# +# .. note:: +# +# * MADIS2NCToolUseCase +# +# Navigate to the :ref:`quick-search` page to discover other similar use cases. +# +# +# +# sphinx_gallery_thumbnail_path = '_static/met_tool_wrapper-MADIS2NC.png' +# diff --git a/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst b/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst new file mode 100644 index 0000000000..767da397da --- /dev/null +++ b/docs/use_cases/met_tool_wrapper/MADIS2NC/README.rst @@ -0,0 +1,2 @@ +MADIS2NC +-------- diff --git a/internal/tests/data/madis/metar_2019040912_F000.nc b/internal/tests/data/madis/metar_2019040912_F000.nc new file mode 100644 index 0000000000..e69de29bb2 diff --git a/internal/tests/data/madis/metar_2019040918_F000.nc b/internal/tests/data/madis/metar_2019040918_F000.nc new file mode 100644 index 0000000000..e69de29bb2 diff --git a/internal/tests/pytests/conftest.py b/internal/tests/pytests/conftest.py index 9853f78fb4..de6a6f4efa 100644 --- a/internal/tests/pytests/conftest.py +++ b/internal/tests/pytests/conftest.py @@ -206,3 +206,35 @@ def get_test_data_path(subdir): return os.path.join(internal_tests_dir, 'data', subdir) return get_test_data_path + + +@pytest.fixture(scope="function") +def compare_command_and_env_vars(): + def do_comparison(all_commands, expected_cmds, env_var_values, wrapper, + special_values=None): + print(f"ALL COMMANDS: {all_commands}") + assert len(all_commands) == len(expected_cmds) + + missing_env = [item for item in env_var_values + if item not in wrapper.WRAPPER_ENV_VAR_KEYS + and item != 'DIAG_ARG'] + env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env + + for (cmd, env_vars), expected_cmd in zip(all_commands, expected_cmds): + # ensure commands are generated as expected + assert cmd == expected_cmd + + # check that environment variables were set properly + # including deprecated env vars (not in wrapper env var keys) + for env_var_key in env_var_keys: + print(f"ENV VAR: {env_var_key}") + match = next((item for item in env_vars if + item.startswith(env_var_key)), None) + assert match is not None + value = match.split('=', 1)[1] + if special_values is not None and env_var_key in special_values: + assert value == special_values[env_var_key] + else: + assert env_var_values.get(env_var_key, '') == value + + return do_comparison diff --git a/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py b/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py index 045195a87c..4209c2e047 100644 --- a/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/ascii2nc/test_ascii2nc_wrapper.py @@ -205,7 +205,8 @@ def test_ascii2nc_missing_inputs(metplus_config, get_test_data_dir, ] ) @pytest.mark.wrapper -def test_ascii2nc_wrapper(metplus_config, config_overrides, env_var_values): +def test_ascii2nc_wrapper(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): wrapper = ascii2nc_wrapper(metplus_config, config_overrides) assert wrapper.isOK @@ -234,30 +235,7 @@ def test_ascii2nc_wrapper(metplus_config, config_overrides, env_var_values): f"-config {config_file} {verbosity}"), ] - assert len(all_commands) == len(expected_cmds) - for (cmd, _), expected_cmd in zip(all_commands, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - env_vars = all_commands[0][1] - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - - assert env_var_values.get(env_var_key, '') == value - - output_base = wrapper.config.getdir('OUTPUT_BASE') - if output_base: - shutil.rmtree(output_base) + compare_command_and_env_vars(all_commands, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py b/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py index d4d1db5858..1f6e9bc7c8 100644 --- a/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/ensemble_stat/test_ensemble_stat_wrapper.py @@ -648,7 +648,7 @@ def test_ensemble_stat_field_info(metplus_config, config_overrides, ) @pytest.mark.wrapper_c def test_ensemble_stat_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -687,31 +687,12 @@ def test_ensemble_stat_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_c diff --git a/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py b/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py index 4d3ca9100d..50a09b569a 100644 --- a/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py +++ b/internal/tests/pytests/wrappers/gen_ens_prod/test_gen_ens_prod_wrapper.py @@ -459,7 +459,7 @@ def test_gen_ens_prod_missing_inputs(metplus_config, get_test_data_dir, allow_mi ) @pytest.mark.wrapper def test_gen_ens_prod_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -508,28 +508,11 @@ def test_gen_ens_prod_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert(cmd == expected_cmd) - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert(match is not None) - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_ENS_FIELD': - assert (actual_value == ens_fmt) - else: - assert(env_var_values.get(env_var_key, '') == actual_value) + special_values = { + 'METPLUS_ENS_FIELD': ens_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py b/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py index b8816ef403..7179109ddc 100644 --- a/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py +++ b/internal/tests/pytests/wrappers/grid_diag/test_grid_diag.py @@ -354,7 +354,8 @@ def test_get_config_file(metplus_config): ] ) @pytest.mark.wrapper -def test_grid_diag(metplus_config, config_overrides, env_var_values): +def test_grid_diag(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -385,25 +386,8 @@ def test_grid_diag(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_DICT': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_DICT': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) diff --git a/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py b/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py index b97ad440e2..8f18f30384 100644 --- a/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/grid_stat/test_grid_stat_wrapper.py @@ -728,7 +728,7 @@ def test_grid_stat_is_prob(metplus_config, config_overrides, expected_values): ) @pytest.mark.wrapper_b def test_grid_stat_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -748,7 +748,6 @@ def test_grid_stat_single_field(metplus_config, config_overrides, for index in range(0, len(run_times)): extra_args[index] += f'-ugrid_config {ugrid_config_file} ' - app_path = os.path.join(config.getdir('MET_BIN_DIR'), wrapper.app_name) verbosity = f"-v {wrapper.c_dict['VERBOSITY']}" config_file = wrapper.c_dict.get('CONFIG_FILE') @@ -764,31 +763,12 @@ def test_grid_stat_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_b diff --git a/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py b/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py index 3845b13ca8..cc49602b90 100644 --- a/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/ioda2nc/test_ioda2nc_wrapper.py @@ -243,7 +243,7 @@ def test_ioda2nc_missing_inputs(metplus_config, get_test_data_dir, missing, ) @pytest.mark.wrapper def test_ioda2nc_wrapper(metplus_config, config_overrides, - env_var_values, extra_args): + env_var_values, extra_args, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -273,25 +273,7 @@ def test_ioda2nc_wrapper(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py b/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py new file mode 100644 index 0000000000..3f3c2622bd --- /dev/null +++ b/internal/tests/pytests/wrappers/madis2nc/test_madis2nc_wrapper.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 + +import pytest + +import os +import shutil + +from metplus.wrappers.madis2nc_wrapper import MADIS2NCWrapper + + +def madis2nc_wrapper(metplus_config, config_overrides=None): + config = metplus_config + overrides = { + 'DO_NOT_RUN_EXE': True, + 'INPUT_MUST_EXIST': False, + 'PROCESS_LIST': 'MADIS2NC', + 'LOOP_BY': 'VALID', + 'VALID_TIME_FMT': '%Y%m%d%H', + 'VALID_BEG': '2019040912', + 'VALID_END': '2019040918', + 'VALID_INCREMENT': '6H', + 'MADIS2NC_INPUT_TEMPLATE': '{INPUT_BASE}/met_test/data/sample_obs/madis/metar_{valid?fmt=%Y%m%d%H}_F000.nc', + 'MADIS2NC_OUTPUT_TEMPLATE': '{OUTPUT_BASE}/madis2nc/metar_{valid?fmt=%Y%m%d%H}.nc', + 'MADIS2NC_CONFIG_FILE': '{PARM_BASE}/met_config/Madis2NcConfig_wrapped', + 'MADIS2NC_TYPE': 'metar', + } + if config_overrides: + for key, value in config_overrides.items(): + overrides[key] = value + + instance = 'overrides' + if not config.has_section(instance): + config.add_section(instance) + for key, value in overrides.items(): + config.set(instance, key, value) + + return MADIS2NCWrapper(config, instance=instance) + + +@pytest.mark.parametrize( + 'missing, run, thresh, errors, allow_missing', [ + (1, 3, 0.5, 0, True), + (1, 3, 0.8, 1, True), + (1, 3, 0.5, 1, False), + ] +) +@pytest.mark.wrapper +def test_madis2nc_missing_inputs(metplus_config, get_test_data_dir, + missing, run, thresh, errors, allow_missing): + config_overrides = { + 'INPUT_MUST_EXIST': True, + 'MADIS2NC_ALLOW_MISSING_INPUTS': allow_missing, + 'MADIS2NC_INPUT_THRESH': thresh, + 'MADIS2NC_INPUT_TEMPLATE': os.path.join(get_test_data_dir('madis'), 'metar_{valid?fmt=%Y%m%d%H}_F000.nc'), + 'VALID_END': '2019041000', + } + wrapper = madis2nc_wrapper(metplus_config, config_overrides) + assert wrapper.isOK + + all_cmds = wrapper.run_all_times() + for cmd, _ in all_cmds: + print(cmd) + + print(f'missing: {wrapper.missing_input_count} / {wrapper.run_count}, errors: {wrapper.errors}') + assert wrapper.missing_input_count == missing + assert wrapper.run_count == run + assert wrapper.errors == errors + + +@pytest.mark.parametrize( + 'config_overrides, env_var_values', [ + ({}, {}), + + ({'MADIS2NC_TIME_SUMMARY_FLAG': 'True'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {flag = TRUE;}'}), + + ({'MADIS2NC_TIME_SUMMARY_RAW_DATA': 'true'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {raw_data = TRUE;}'}), + + ({'MADIS2NC_TIME_SUMMARY_BEG': '123456'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {beg = "123456";}'}), + + ({'MADIS2NC_TIME_SUMMARY_END': '123456'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {end = "123456";}'}), + + ({'MADIS2NC_TIME_SUMMARY_STEP': '500'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {step = 500;}'}), + + ({'MADIS2NC_TIME_SUMMARY_WIDTH': '900'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {width = 900;}'}), + # width as dictionary + ({'MADIS2NC_TIME_SUMMARY_WIDTH': '{ beg = -21600; end = 0; }'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {width = { beg = -21600; end = 0; };}'}), + + ({'MADIS2NC_TIME_SUMMARY_GRIB_CODE': '12, 203, 212'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {grib_code = [12, 203, 212];}'}), + + ({'MADIS2NC_TIME_SUMMARY_OBS_VAR': 'TMP, HGT, PRES'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {obs_var = ["TMP", "HGT", "PRES"];}'}), + + ({'MADIS2NC_TIME_SUMMARY_TYPE': 'min, range, max'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {type = ["min", "range", "max"];}'}), + + ({'MADIS2NC_TIME_SUMMARY_VLD_FREQ': '2'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {vld_freq = 2;}'}), + + ({'MADIS2NC_TIME_SUMMARY_VLD_THRESH': '0.5'}, + {'METPLUS_TIME_SUMMARY_DICT': 'time_summary = {vld_thresh = 0.5;}'}), + + ({'MADIS2NC_TIME_SUMMARY_FLAG': 'false', + 'MADIS2NC_TIME_SUMMARY_RAW_DATA': 'false', + 'MADIS2NC_TIME_SUMMARY_BEG': '123456', + 'MADIS2NC_TIME_SUMMARY_END': '125634', + 'MADIS2NC_TIME_SUMMARY_STEP': '500', + 'MADIS2NC_TIME_SUMMARY_WIDTH': '900', + 'MADIS2NC_TIME_SUMMARY_GRIB_CODE': '12, 203, 212', + 'MADIS2NC_TIME_SUMMARY_OBS_VAR': 'TMP, HGT, PRES', + 'MADIS2NC_TIME_SUMMARY_TYPE': 'min, range, max', + 'MADIS2NC_TIME_SUMMARY_VLD_FREQ': '2', + 'MADIS2NC_TIME_SUMMARY_VLD_THRESH': '0.5', + }, + {'METPLUS_TIME_SUMMARY_DICT': + ('time_summary = {flag = FALSE;raw_data = FALSE;beg = "123456";' + 'end = "125634";step = 500;width = 900;' + 'grib_code = [12, 203, 212];obs_var = ["TMP", "HGT", "PRES"];' + 'type = ["min", "range", "max"];' + 'vld_freq = 2;vld_thresh = 0.5;}')}), + ({'MADIS2NC_QC_DD': '4,5,6'}, {}), + ({'MADIS2NC_LVL_DIM': 'P500,P750'}, {}), + ({'MADIS2NC_REC_BEG': '2'}, {}), + ({'MADIS2NC_REC_END': '3'}, {}), + ({'MADIS2NC_MASK_GRID': 'mask_grid'}, {}), + ({'MADIS2NC_MASK_POLY': '/some/path/to/mask/poly'}, {}), + ({'MADIS2NC_MASK_SID': 'mask_sid,/some/path/to/mask/sid'}, {}), + ({'MADIS2NC_QC_DD': '4,5,6', + 'MADIS2NC_LVL_DIM': 'P500,P750', + 'MADIS2NC_REC_BEG': '2', + 'MADIS2NC_REC_END': '3', + 'MADIS2NC_MASK_GRID': 'mask_grid', + 'MADIS2NC_MASK_POLY': '/some/path/to/mask/poly', + 'MADIS2NC_MASK_SID': 'mask_sid,/some/path/to/mask/sid'}, {}), + ({'MADIS2NC_TIME_OFFSET_WARNING': '4', }, + {'METPLUS_TIME_OFFSET_WARNING': 'time_offset_warning = 4;'}), + + ] +) +@pytest.mark.wrapper +def test_madis2nc_wrapper(metplus_config, config_overrides, + env_var_values, compare_command_and_env_vars): + wrapper = madis2nc_wrapper(metplus_config, config_overrides) + assert wrapper.isOK + + input_dir = os.path.dirname(wrapper.config.getraw('config', 'MADIS2NC_INPUT_TEMPLATE')) + input_file1 = 'metar_2019040912_F000.nc' + input_file2 = 'metar_2019040918_F000.nc' + + output_dir = os.path.dirname(wrapper.config.getraw('config', 'MADIS2NC_OUTPUT_TEMPLATE')) + output_file1 = 'metar_2019040912.nc' + output_file2 = 'metar_2019040918.nc' + + all_commands = wrapper.run_all_times() + print(f"ALL COMMANDS: {all_commands}") + + app_path = os.path.join(wrapper.config.getdir('MET_BIN_DIR'), + wrapper.app_name) + verbosity = f"-v {wrapper.c_dict['VERBOSITY']}" + config_file = wrapper.c_dict.get('CONFIG_FILE') + + in_type = config_overrides['MADIS2NC_TYPE'] if 'MADIS2NC_TYPE' in config_overrides else 'metar' + extra_args = '' + for optional_arg in ('qc_dd', 'lvl_dim', 'rec_beg', 'rec_end', 'mask_grid', 'mask_poly', 'mask_sid'): + if f'MADIS2NC_{optional_arg.upper()}' in config_overrides: + extra_args += f" -{optional_arg} {config_overrides[f'MADIS2NC_{optional_arg.upper()}']}" + + expected_cmds = [ + (f"{app_path} {verbosity} {input_dir}/{input_file1} {output_dir}/{output_file1} " + f"-type {in_type} -config {config_file}{extra_args}"), + (f"{app_path} {verbosity} {input_dir}/{input_file2} {output_dir}/{output_file2} " + f"-type {in_type} -config {config_file}{extra_args}"), + ] + + compare_command_and_env_vars(all_commands, expected_cmds, env_var_values, wrapper) + + +@pytest.mark.wrapper +def test_get_config_file(metplus_config): + fake_config_name = '/my/config/file' + config = metplus_config + default_config_file = os.path.join(config.getdir('PARM_BASE'), + 'met_config', + 'Madis2NcConfig_wrapped') + + wrapper = MADIS2NCWrapper(config) + assert wrapper.c_dict['CONFIG_FILE'] == default_config_file + + config.set('config', 'MADIS2NC_CONFIG_FILE', fake_config_name) + wrapper = MADIS2NCWrapper(config) + assert wrapper.c_dict['CONFIG_FILE'] == fake_config_name diff --git a/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py b/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py index 696b1921c1..f64953e87a 100644 --- a/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py +++ b/internal/tests/pytests/wrappers/mode/test_mode_wrapper.py @@ -390,7 +390,8 @@ def test_mode_missing_inputs(metplus_config, get_test_data_dir, ] ) @pytest.mark.wrapper_a -def test_mode_single_field(metplus_config, config_overrides, env_var_values): +def test_mode_single_field(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config # set config variables needed to run @@ -418,7 +419,6 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") # set default values in expected output list # only if they are not set and if MODE_GRID_RES is set @@ -442,28 +442,12 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): f'{met_name} = {default_val};' ) - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py b/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py index a3057838f8..97c9b6bd6c 100644 --- a/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py +++ b/internal/tests/pytests/wrappers/mtd/test_mtd_wrapper.py @@ -209,7 +209,8 @@ def test_mtd_missing_inputs(metplus_config, get_test_data_dir, ] ) @pytest.mark.wrapper -def test_mode_single_field(metplus_config, config_overrides, env_var_values): +def test_mode_single_field(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = metplus_config # set config variables needed to run @@ -237,29 +238,12 @@ def test_mode_single_field(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py index dc7a617179..1622bb6c9a 100644 --- a/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py +++ b/internal/tests/pytests/wrappers/pb2nc/test_pb2nc_wrapper.py @@ -291,7 +291,8 @@ def test_find_input_files(metplus_config, offsets, offset_to_find): ] ) @pytest.mark.wrapper -def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values): +def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): input_dir = '/some/input/dir' config = metplus_config @@ -348,24 +349,7 @@ def test_pb2nc_all_fields(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py b/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py index 9589448f2f..e38d84400a 100755 --- a/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/point_stat/test_point_stat_wrapper.py @@ -27,6 +27,7 @@ ugrid_config_file = '/some/path/UgridConfig_fake' + def set_minimum_config_settings(config): # set config variables to prevent command from running and bypass check # if input files actually exist @@ -678,7 +679,7 @@ def test_met_dictionary_in_var_options(metplus_config): ) @pytest.mark.wrapper_a def test_point_stat_all_fields(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): level_no_quotes = '(*,*)' level_with_quotes = f'"{level_no_quotes}"' @@ -775,35 +776,16 @@ def test_point_stat_all_fields(metplus_config, config_overrides, f"{config_file}{extra_args[index]}-outdir {out_dir}/{valids[index]}" ) - all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - fcst_fmt = f"field = [{','.join(fcst_fmts)}];" obs_fmt = f"field = [{','.join(obs_fmts)}];" - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == value + all_cmds = wrapper.run_all_times() + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_a diff --git a/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py b/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py index 5589b054a1..dd69b8e4eb 100644 --- a/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py +++ b/internal/tests/pytests/wrappers/series_analysis/test_series_analysis.py @@ -424,7 +424,7 @@ def test_series_analysis_missing_inputs(metplus_config, get_test_data_dir, ) @pytest.mark.wrapper_a def test_series_analysis_single_field(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config @@ -464,7 +464,6 @@ def test_series_analysis_single_field(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") expected_len = len(expected_cmds) if 'SERIES_ANALYSIS_GENERATE_PLOTS' in config_overrides: @@ -473,29 +472,15 @@ def test_series_analysis_single_field(metplus_config, config_overrides, expected_len += 4 assert len(all_cmds) == expected_len - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - elif env_var_key == 'METPLUS_OUTPUT_STATS_DICT' and 'METPLUS_OUTPUT_STATS_DICT' not in env_var_values: - assert actual_value == stat_list_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + if 'METPLUS_OUTPUT_STATS_DICT' not in env_var_values: + special_values['METPLUS_OUTPUT_STATS_DICT'] = stat_list_fmt + # only compare first command since the rest are not series_analysis + compare_command_and_env_vars(all_cmds[0:1], expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_a diff --git a/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py b/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py index d0601dcb4a..6d0f619ec7 100644 --- a/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py +++ b/internal/tests/pytests/wrappers/stat_analysis/test_stat_analysis.py @@ -18,6 +18,7 @@ JOB_ARGS = '-job filter' + def stat_analysis_wrapper(metplus_config): """! Returns a default StatAnalysisWrapper with /path/to entries in the metplus_system.conf and metplus_runtime.conf configuration diff --git a/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py b/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py index 19813070cf..88b9ef1eb5 100644 --- a/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_diag/test_tc_diag_wrapper.py @@ -61,7 +61,6 @@ def set_minimum_config_settings(config): config.set('config', 'BOTH_VAR2_LEVELS', 'P1000, P900, P800, P700, P500, P100') - @pytest.mark.parametrize( 'missing, run, thresh, errors, allow_missing', [ (1, 3, 0.5, 0, True), @@ -250,7 +249,7 @@ def test_tc_diag_missing_inputs(metplus_config, get_test_data_dir, ) @pytest.mark.wrapper def test_tc_diag_run(metplus_config, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -284,28 +283,11 @@ def test_tc_diag_run(metplus_config, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_FIELD': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_FIELD': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py b/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py index 9051d682d7..c9beca4fde 100644 --- a/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_gen/test_tc_gen_wrapper.py @@ -348,7 +348,8 @@ def test_tc_gen_missing_inputs(metplus_config, get_test_data_dir, allow_missing, ] ) @pytest.mark.wrapper_a -def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, env_var_values): +def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, + env_var_values, compare_command_and_env_vars): # expected number of 2016 files (including file_list line) expected_genesis_count = 7 expected_track_count = expected_genesis_count @@ -440,25 +441,7 @@ def test_tc_gen(metplus_config, get_test_data_dir, config_overrides, env_var_val ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) # verify file count of genesis, edeck, shape, and track file list files with open(genesis_path, 'r') as file_handle: diff --git a/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py b/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py index 543fce479b..70adbd62d5 100644 --- a/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_pairs/test_tc_pairs_wrapper.py @@ -617,7 +617,7 @@ def test_tc_pairs_storm_id_lists(metplus_config, get_test_data_dir, config_overr ) @pytest.mark.wrapper def test_tc_pairs_run(metplus_config, get_test_data_dir, loop_by, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config remove_beg = remove_end = remove_match_points = False @@ -684,26 +684,7 @@ def test_tc_pairs_run(metplus_config, get_test_data_dir, loop_by, config_overrid ) all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS - and item != 'DIAG_ARG'] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) if remove_beg: del env_var_values[f'METPLUS_{loop_by}_BEG'] diff --git a/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py b/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py index b027050510..0f52648e6b 100644 --- a/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py +++ b/internal/tests/pytests/wrappers/tc_stat/test_tc_stat_wrapper.py @@ -251,7 +251,8 @@ def test_tc_stat_handle_jobs(metplus_config, config_overrides, expected_dirs, ] ) @pytest.mark.wrapper -def test_tc_stat_run(metplus_config, config_overrides, env_var_values): +def test_tc_stat_run(metplus_config, config_overrides, env_var_values, + compare_command_and_env_vars): config = get_config(metplus_config) # set config variable overrides @@ -292,25 +293,7 @@ def test_tc_stat_run(metplus_config, config_overrides, env_var_values): ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - assert env_var_values.get(env_var_key, '') == actual_value + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, wrapper) @pytest.mark.parametrize( diff --git a/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py b/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py index e6981ddcd1..9961b78277 100644 --- a/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py +++ b/internal/tests/pytests/wrappers/tcrmw/test_tcrmw_wrapper.py @@ -135,7 +135,7 @@ def set_minimum_config_settings(config): ) @pytest.mark.wrapper def test_tc_rmw_run(metplus_config, get_test_data_dir, config_overrides, - env_var_values): + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -171,28 +171,11 @@ def test_tc_rmw_run(metplus_config, get_test_data_dir, config_overrides, ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - assert len(all_cmds) == len(expected_cmds) - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - for env_var_key in env_var_keys: - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - print(f'Checking env var: {env_var_key}') - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_DATA_FIELD': - assert actual_value == data_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_DATA_FIELD': data_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper diff --git a/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py b/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py index 97b364c7a0..52e4528ad2 100644 --- a/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py +++ b/internal/tests/pytests/wrappers/wavelet_stat/test_wavelet_stat.py @@ -322,7 +322,8 @@ def test_wavelet_stat_is_prob(metplus_config, config_overrides, expected_values) ] ) @pytest.mark.wrapper_b -def test_wavelet_stat_single_field(metplus_config, config_overrides, env_var_values): +def test_wavelet_stat_single_field(metplus_config, config_overrides, + env_var_values, compare_command_and_env_vars): config = metplus_config set_minimum_config_settings(config) @@ -349,31 +350,12 @@ def test_wavelet_stat_single_field(metplus_config, config_overrides, env_var_val ] all_cmds = wrapper.run_all_times() - print(f"ALL COMMANDS: {all_cmds}") - - missing_env = [item for item in env_var_values - if item not in wrapper.WRAPPER_ENV_VAR_KEYS] - env_var_keys = wrapper.WRAPPER_ENV_VAR_KEYS + missing_env - - assert len(all_cmds) == len(expected_cmds) - for (cmd, env_vars), expected_cmd in zip(all_cmds, expected_cmds): - # ensure commands are generated as expected - assert cmd == expected_cmd - - # check that environment variables were set properly - # including deprecated env vars (not in wrapper env var keys) - for env_var_key in env_var_keys: - print(f"ENV VAR: {env_var_key}") - match = next((item for item in env_vars if - item.startswith(env_var_key)), None) - assert match is not None - actual_value = match.split('=', 1)[1] - if env_var_key == 'METPLUS_FCST_FIELD': - assert actual_value == fcst_fmt - elif env_var_key == 'METPLUS_OBS_FIELD': - assert actual_value == obs_fmt - else: - assert env_var_values.get(env_var_key, '') == actual_value + special_values = { + 'METPLUS_FCST_FIELD': fcst_fmt, + 'METPLUS_OBS_FIELD': obs_fmt, + } + compare_command_and_env_vars(all_cmds, expected_cmds, env_var_values, + wrapper, special_values) @pytest.mark.wrapper_b diff --git a/internal/tests/use_cases/all_use_cases.txt b/internal/tests/use_cases/all_use_cases.txt index daef1a124e..38c743911a 100644 --- a/internal/tests/use_cases/all_use_cases.txt +++ b/internal/tests/use_cases/all_use_cases.txt @@ -63,6 +63,7 @@ Category: met_tool_wrapper 61::PlotPointObs:: met_tool_wrapper/PlotPointObs/PlotPointObs.conf 62::TCDiag:: met_tool_wrapper/TCDiag/TCDiag.conf 63::WaveletStat:: met_tool_wrapper/WaveletStat/WaveletStat.conf +64::MADIS2NC:: met_tool_wrapper/MADIS2NC/MADIS2NC.conf Category: air_quality_and_comp 0::EnsembleStat_fcstICAP_obsMODIS_aod::model_applications/air_quality_and_comp/EnsembleStat_fcstICAP_obsMODIS_aod.conf diff --git a/metplus/util/constants.py b/metplus/util/constants.py index 72a316281f..3208aa2a26 100644 --- a/metplus/util/constants.py +++ b/metplus/util/constants.py @@ -20,7 +20,7 @@ 'griddiag': 'GridDiag', 'gridstat': 'GridStat', 'ioda2nc': 'IODA2NC', - 'makeplots': 'MakePlots', + 'madis2nc': 'MADIS2NC', 'metdbload': 'METDbLoad', 'mode': 'MODE', 'mtd': 'MTD', diff --git a/metplus/wrappers/command_builder.py b/metplus/wrappers/command_builder.py index 7cde480858..cc5d6aedc9 100755 --- a/metplus/wrappers/command_builder.py +++ b/metplus/wrappers/command_builder.py @@ -1576,15 +1576,20 @@ def handle_time_summary_dict(self): 'step': 'int', 'width': ('string', 'remove_quotes'), 'grib_code': ('list', 'remove_quotes,allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_GRIB_CODES']), + [f'{app_upper}_TIME_SUMMARY_GRIB_CODE', + f'{app_upper}_TIME_SUMMARY_GRIB_CODES']), 'obs_var': ('list', 'allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_VAR_NAMES']), + [f'{app_upper}_TIME_SUMMARY_OBS_VAR', + f'{app_upper}_TIME_SUMMARY_VAR_NAMES']), 'type': ('list', 'allow_empty', None, - [f'{app_upper}_TIME_SUMMARY_TYPES']), + [f'{app_upper}_TIME_SUMMARY_TYPE', + f'{app_upper}_TIME_SUMMARY_TYPES']), 'vld_freq': ('int', None, None, - [f'{app_upper}_TIME_SUMMARY_VALID_FREQ']), + [f'{app_upper}_TIME_SUMMARY_VLD_FREQ', + f'{app_upper}_TIME_SUMMARY_VALID_FREQ']), 'vld_thresh': ('float', None, None, - [f'{app_upper}_TIME_SUMMARY_VALID_THRESH']), + [f'{app_upper}_TIME_SUMMARY_VLD_THRESH', + f'{app_upper}_TIME_SUMMARY_VALID_THRESH']), }) def handle_mask(self, single_value=False, get_flags=False, get_point=False): diff --git a/metplus/wrappers/madis2nc_wrapper.py b/metplus/wrappers/madis2nc_wrapper.py new file mode 100755 index 0000000000..420a5d4f22 --- /dev/null +++ b/metplus/wrappers/madis2nc_wrapper.py @@ -0,0 +1,127 @@ +""" +Program Name: madis2nc_wrapper.py +Contact(s): George McCabe +Abstract: Builds command for and runs madis2nc +History Log: Initial version +Usage: +Parameters: None +Input Files: MADIS files +Output Files: nc files +Condition codes: 0 for success, 1 for failure +""" + +import os + +from ..util import do_string_sub, MISSING_DATA_VALUE +from . import RuntimeFreqWrapper + +'''!@namespace MADIS2NCWrapper +@brief Wraps the madis2nc tool to reformat MADIS format to NetCDF +@endcode +''' + + +class MADIS2NCWrapper(RuntimeFreqWrapper): + + RUNTIME_FREQ_DEFAULT = 'RUN_ONCE_FOR_EACH' + RUNTIME_FREQ_SUPPORTED = 'ALL' + + WRAPPER_ENV_VAR_KEYS = [ + 'METPLUS_TIME_SUMMARY_DICT', + ] + + def __init__(self, config, instance=None): + self.app_name = "madis2nc" + self.app_path = os.path.join(config.getdir('MET_BIN_DIR', ''), + self.app_name) + super().__init__(config, instance=instance) + + def create_c_dict(self): + c_dict = super().create_c_dict() + c_dict['VERBOSITY'] = self.config.getstr('config', + 'LOG_MADIS2NC_VERBOSITY', + c_dict['VERBOSITY']) + + # file I/O + c_dict['ALLOW_MULTIPLE_FILES'] = True + self.get_input_templates(c_dict, { + 'OBS': {'prefix': 'MADIS2NC', 'required': True}, + }) + + c_dict['OUTPUT_DIR'] = self.config.getdir('MADIS2NC_OUTPUT_DIR', '') + c_dict['OUTPUT_TEMPLATE'] = ( + self.config.getraw('config', 'MADIS2NC_OUTPUT_TEMPLATE') + ) + if not c_dict['OUTPUT_TEMPLATE']: + self.log_error('MADIS2NC_OUTPUT_TEMPLATE must be set') + + # config file settings + c_dict['CONFIG_FILE'] = self.get_config_file('Madis2NcConfig_wrapped') + self.handle_time_summary_dict() + + # command line settings + c_dict['TYPE'] = self.config.getraw('config', 'MADIS2NC_TYPE') + if not c_dict['TYPE']: + self.log_error('Must set MADIS2NC_TYPE') + + c_dict['QC_DD'] = self.config.getraw('config', 'MADIS2NC_QC_DD') + c_dict['LVL_DIM'] = self.config.getraw('config', 'MADIS2NC_LVL_DIM') + c_dict['REC_BEG'] = self.config.getint('config', 'MADIS2NC_REC_BEG') + if c_dict['REC_BEG'] == MISSING_DATA_VALUE: + c_dict['REC_BEG'] = '' + c_dict['REC_END'] = self.config.getint('config', 'MADIS2NC_REC_END') + if c_dict['REC_END'] == MISSING_DATA_VALUE: + c_dict['REC_END'] = '' + c_dict['MASK_GRID'] = self.config.getraw('config', 'MADIS2NC_MASK_GRID') + c_dict['MASK_POLY'] = self.config.getraw('config', 'MADIS2NC_MASK_POLY') + c_dict['MASK_SID'] = self.config.getraw('config', 'MADIS2NC_MASK_SID') + + return c_dict + + def get_command(self): + """!Build command to run madis2nc + + @returns str: madis2nc command + """ + return (f"{self.app_path} -v {self.c_dict['VERBOSITY']}" + f" {' '.join(self.infiles)} {self.get_output_path()}" + f" {' '.join(self.args)}") + + def find_input_files(self, time_info): + """!Get list of input files to pass to command. Sets self.infiles. + + @param time_info dictionary containing time information + @returns bool: True if files were found, False otherwise + """ + if not self.c_dict.get('ALL_FILES'): + return False + + input_files = self.c_dict['ALL_FILES'][0].get('OBS', []) + if not input_files: + return False + + self.logger.debug(f"Adding input: {' and '.join(input_files)}") + self.infiles.extend(input_files) + return True + + def set_command_line_arguments(self, time_info): + """!Read self.c_dict and set command line arguments in self.args. + + @param time_info dictionary containing time information + """ + # set required command line arguments + val = do_string_sub(self.c_dict['TYPE'], **time_info) + self.args.append(f"-type {val}") + + config_file = do_string_sub(self.c_dict['CONFIG_FILE'], **time_info) + self.args.append(f"-config {config_file}") + + # set optional command line arguments if specified + optional_args = ('qc_dd', 'lvl_dim', 'rec_beg', 'rec_end', 'mask_grid', + 'mask_poly', 'mask_sid') + for arg in optional_args: + if self.c_dict[arg.upper()]: + val = self.c_dict[arg.upper()] + if isinstance(val, str): + val = do_string_sub(val, **time_info) + self.args.append(f"-{arg} {val}") diff --git a/parm/met_config/Madis2NcConfig_wrapped b/parm/met_config/Madis2NcConfig_wrapped new file mode 100644 index 0000000000..282f6782f1 --- /dev/null +++ b/parm/met_config/Madis2NcConfig_wrapped @@ -0,0 +1,30 @@ +//////////////////////////////////////////////////////////////////////////////// +// +// MADIS2NC configuration file. +// +// For additional information, please see the MET Users Guide. +// +//////////////////////////////////////////////////////////////////////////////// + +// +// The parameters listed below are used to summarize the MADIS data read in +// + +// +// Time periods for the summarization +// obs_var (string array) is added and works like grib_code (int array) +// when the obs name is given instead of grib_code +// + +//time_summary = { +${METPLUS_TIME_SUMMARY_DICT} + +// +// Indicate a version number for the contents of this configuration file. +// The value should generally not be modified. +// +//version = "V12.0.0"; + +${METPLUS_TIME_OFFSET_WARNING} + +${METPLUS_MET_CONFIG_OVERRIDES} diff --git a/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf b/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf new file mode 100644 index 0000000000..ac4067dd6c --- /dev/null +++ b/parm/use_cases/met_tool_wrapper/MADIS2NC/MADIS2NC.conf @@ -0,0 +1,88 @@ +[config] + +# Documentation for this use case can be found at +# https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/MADIS2NC/MADIS2NC.html + +# For additional information, please see the METplus Users Guide. +# https://metplus.readthedocs.io/en/latest/Users_Guide + +### +# Processes to run +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list +### + +PROCESS_LIST = MADIS2NC + + +### +# Time Info +# LOOP_BY options are INIT, VALID, RETRO, and REALTIME +# If set to INIT or RETRO: +# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set +# If set to VALID or REALTIME: +# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set +# LEAD_SEQ is the list of forecast leads to process +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control +### + +LOOP_BY = INIT +INIT_TIME_FMT = %Y%m%d%H +INIT_BEG = 2012040912 +INIT_END = 2012040912 +INIT_INCREMENT = 1H + +LEAD_SEQ = 0 + +MADIS2NC_RUNTIME_FREQ = RUN_ONCE_FOR_EACH + + +### +# File I/O +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info +### + +MADIS2NC_INPUT_DIR = +MADIS2NC_INPUT_TEMPLATE = {INPUT_BASE}/met_test/data/sample_obs/madis/metar/metar_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc + +MADIS2NC_OUTPUT_DIR = +MADIS2NC_OUTPUT_TEMPLATE = {OUTPUT_BASE}/madis2nc/met_metar_{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}.nc + +#MADIS2NC_SKIP_IF_OUTPUT_EXISTS = False + +#MADIS2NC_FILE_WINDOW_BEGIN = 0 +#MADIS2NC_FILE_WINDOW_END = 0 + +### +# MADIS2NC Settings +# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#madis2nc +### + +#LOG_MADIS2NC_VERBOSITY = 3 + +MADIS2NC_CONFIG_FILE = {PARM_BASE}/met_config/Madis2NcConfig_wrapped + +MADIS2NC_TYPE = metar + +#MADIS2NC_QC_DD = +#MADIS2NC_LVL_DIM = +#MADIS2NC_REC_BEG = +#MADIS2NC_REC_END = +#MADIS2NC_MASK_GRID = +#MADIS2NC_MASK_POLY = +#MADIS2NC_MASK_SID = + +#MADIS2NC_TIME_SUMMARY_FLAG = +#MADIS2NC_TIME_SUMMARY_RAW_DATA = +#MADIS2NC_TIME_SUMMARY_BEG = +#MADIS2NC_TIME_SUMMARY_END = +#MADIS2NC_TIME_SUMMARY_STEP = +#MADIS2NC_TIME_SUMMARY_WIDTH = +#MADIS2NC_TIME_SUMMARY_GRIB_CODE = +#MADIS2NC_TIME_SUMMARY_OBS_VAR = +#MADIS2NC_TIME_SUMMARY_TYPE = +#MADIS2NC_TIME_SUMMARY_VLD_FREQ = +#MADIS2NC_TIME_SUMMARY_VLD_THRESH = + +#MADIS2NC_TIME_OFFSET_WARNING = + +#MADIS2NC_MET_CONFIG_OVERRIDES = \ No newline at end of file