Enhance the MET point processing tools to read the Python 'point_data' variable instead of just 'met_point_data'.

[JohnHalleyGotway]

Describe the Enhancement

While working on an example of python embedding of point observations in Point-Stat, @j-opatz had a difficult time storing buoy data in the met_point_data python data structure defined in section 35.5 of the user's guide. He and I spent a 1.5 hours massaging the data to make it work and realize that doing so for the average user would likely be impossible.

Note that section 35.5 defines both the point_data ascii convention expected by the ascii2nc tool and the met_point_data convention expected by the other tools. The former is much easier to use than the latter.

This issue is to enhance MET's python embedding of point observations to support either variation in all the tools. If the point_data object exists, extract and process the data as such. If the met_point_data object exists, do the same.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2793521 (was 2773542 R2O)

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus Enhancement: Update use cases to use new Python directory structure in MET METplus#2115

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[DanielAdriaansen]

One question- it looks like the implementation of this removes the possibility of providing point_data to all MET tools, is that correct? The way forward is to always require met_point_data, but a helper function called convert_point_data exists for backwards compatibility for now?

Or is it possible to provide either point_data or met_point_data and MET will decide which exists?

[JohnHalleyGotway]

The latter. Ideally it'd be possible to provide either point_data or met_point_data and MET will see which exists and use it accordingly.

The first option would still be useful. However, it'll raise the bar for users, requiring them to understand the details, find and use the helper function, and for use to write more documentation and users to read it.

[DanielAdriaansen]

Ideally it'd be possible to provide either point_data or met_point_data and MET will see which exists and use it accordingly.

I guess what I am saying, is that from reviewing what @hsoh-u did, I don't see how it's possible to provide point_data any longer. It looks like met_point_data is required, as evidenced by the call to convert_point_data inside read_ascii_point.py:

MET/scripts/python/read_ascii_point.py

Lines 37 to 43 in 911ac9b

	point_data = pd.read_csv(input_file, header=None, delim_whitespace=True, keep_default_na=False,
	names=['typ', 'sid', 'vld', 'lat', 'lon', 'elv', 'var', 'lvl', 'hgt', 'qc', 'obs'],
	dtype={'typ':'str', 'sid':'str', 'vld':'str', 'var':'str', 'qc':'str'}).values.tolist()
	print(" point_data: Data Length:\t" + repr(len(point_data)))
	print(" point_data: Data Type:\t" + repr(type(point_data)))
	met_point_data = convert_point_data(point_data)
	print(" met_point_data: Data Type:\t" + repr(type(met_point_data)))

If users have any Python Embedding scripts that use point_data, they will have to:

1. Add this line:
   from met_point_obs import convert_point_data
2. Add this line:
   met_point_data = convert_point_data(point_data)

It is possible though, that I am not understanding the changes fully and it would be possible to still provide point_data by itself. @hsoh-u can you confirm? Is point_data still supported? Or do users have to modify their existing code to transform to met_point_data?

[DanielAdriaansen]

OK, @hsoh-u redirected me to #2302, which is what #2340 is closing, and NOT this issue. Thanks Howard!

I was the creator of #2302, which we decided would be "bridge" work to get us to fully implementing what's desired in this issue (#2285).

@JohnHalleyGotway it sounds like we will get to this issue at a later date which is supporting either point_data or met_point_data in all MET tools regardless of what the user provides.