Create the new Gen-Ens-Prod tool for ensemble product generation.

[JohnHalleyGotway]

Describe the New Feature

Create a new tool for ensemble product generation named Gen-Ens-Prod. This tool requires a configuration file and should contain the ensemble product generation currently performed for the fields in the "ens" dictionary of Ensemble-Stat. Consider renaming the "ens" dictionary to "data" to be consistent with the conventions of the Grid-Diag tool.

This tool does not process observations, but must support climatology mean and standard deviation data to support the use of climatological distribution percentile thresholds (e.g. >CDP75).

Acceptance Testing

Ensure that other MET tools can read the NetCDF output created by this tool.

Time Estimate

2 weeks.

Sub-Issues

Consider breaking the new feature down into sub-issues.
Sub-issues will likely be required. Not sure what they are yet.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Split 2793541, 2700041, 2799991

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @JohnHalleyGotway
• Select scientist(s) or no scientist required: @j-opatz
  Can also solicit feedback from @mpm-meto, @jwolff-ncar, and @michelleharrold.

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
• New Wrapper: GenEnsProd METplus#1180
• No downstream impacts on METdatadb, METplotpy, METcalcpy, METviewer, or METexpress.

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

Progress. I have an initial version of gen_ens_prod working in my local feature_1904_gen_ens_prod branch. In Ensemble-Stat, the ensemble_flag dictionary, can only be set once and is applied to all the fields in the "ens" dictionary. So all products are derived for all fields. Recommend that we enable the ensemble_flag to be specified separately for each "ens" field so that users can control the output separately for each field. Here's the usage statement:

Usage: gen_ens_prod
	-ens file_1 ... file_n | ens_file_list
	-out file
	-config file
	[-ctrl file]
	[-log file]
	[-v level]

	where	"-ens file_1 ... file_n" are the gridded ensemble data files to be used (required).
		"ens_file_list" is an ASCII file containing a list of ensemble member file names (required).
		"-out file" is the NetCDF output file for the derived ensemble products (required).
		"-config file" is a GenEnsProdConfig file containing the desired configuration settings (required).
		"-ctrl file" is the gridded ensemble control data file included in mean but excluded from the spread (optional).
		"-log file" outputs log messages to the specified file (optional).
		"-v level" overrides the default level of logging (2) (optional).

[JohnHalleyGotway]

List of enhancements to make:

• DONE 9/23/2021: Let ensemble_flag be specified separately for each ens.field array entry.

• DONE 9/24/2021: Add support for climo and climo_cdp options in the ensemble_flag:

ensemble_flag = {
...
   climo     = TRUE;
   climo_cdp = TRUE;
...
}

Since this tool can read climo data and use them to define climo distribution percentile thresholds (e.g. >CDP50), we should be able to write these fields to the output, as Grid-Stat does.

• DONE 9/24/2021: Update logic for handling the ensemble control member. If the -ctrl option is used, the tool reads all the ensemble fields from it. So that assumes the control member is the same file format as the other members. The only special logic is that the control member is excluded from the computation of spread, but it's used for everything else.

• DONE 9/28/21: Update the gen_ens_prod documentation.

• DONE 9/27/21: Add unit_gen_ens_prod.xml unit tests. Call the tool twice, once with/without the -cntrl option. Confirm that the ensemble spreads match but means differ.

• [XXX] Write NetCDF output following the CF convention rather than MET's internal format. Consider moving this into a new GitHub issue instead.

NOTE Will not make this change as part of this issue. The changes required would be mostly in the common library code and a piecemeal approach isn't a good idea here. Instead make these changes as part of #660.