Merge pull request #2340 from dtcenter/feature_2285_python_embedding_…

…converting

Feature 2302 python embedding converting

docs/Users_Guide/appendixF.rst

@@ -250,54 +250,29 @@ The Ensemble-Stat, Series-Analysis, and MTD tools support the use of file lists
 Python Embedding for Point Observations
 =======================================
 
-The ASCII2NC tool supports the "-format python" option. With this option, point observations may be passed as input. An example of this is provided in :numref:`ascii2nc-pyembed`. That example uses the **read_ascii_point.py** sample script which is included with the MET code. It reads ASCII data in MET's 11-column point observation format and stores it in a Pandas dataframe to be read by the ASCII2NC tool with Python.
+The ASCII2NC tool supports the "-format python" option. With this option, point observations may be passed as input. An example of this is shown below:
 
-The **read_ascii_point.py** sample script can be found in:
-
-• MET installation directory in *MET_BASE/python*.
+.. code-block:: none
 
-• `MET GitHub repository <https://github.com/dtcenter/MET>`_ in *met/scripts/python*.
+                ascii2nc -format python \
+                "MET_BASE/python/read_ascii_point.py sample_ascii_obs.txt" \
+                sample_ascii_obs_python.nc
 
-The Point2Grid, Plot-Point-Obs, Ensemble-Stat, and Point-Stat tools also process point observations. They support python embedding of point observations directly on the command line by replacing the input MET NetCDF point observation file name with the python command to be run. The command must begin with the prefix 'PYTHON_NUMPY=' and be followed by the path to python script and any arguments. The full command should be enclosed in single quotes to prevent embedded whitespace from causing parsing errors. The customized python script is expected to extend MET_BASE/python/met_point_obs.py. That script creates a python variable named **met_point_data** which is a dictionary containing formatted point observation data.
+The Point2Grid, Plot-Point-Obs, Ensemble-Stat, and Point-Stat tools also process point observations. They support POython embedding of point observations directly on the command line by replacing the input MET NetCDF point observation file name with the Python command to be run. The Python command must begin with the prefix 'PYTHON_NUMPY=' and be followed by the path to the User's Python script and any arguments. The full command should be enclosed in single quotes to prevent embedded whitespace from causing parsing errors. An example of this is shown below:
 
 .. code-block:: none
 
-  met_point_data = {
-
-     'use_var_id':  True/False,     # obs_vid are variable index if True, otherwise GRIB codes
-
-     # Header data
-     'nhdr':        integer_value,  # number of headers
-     'pbhdr':       integer_value,  # number of PREPBUFR specific headers
-     'nhdr_typ':    integer_value,  # number of message types
-     'nhdr_sid':    integer_value,  # number of station IDs
-     'nhdr_vld':    integer_value,  # number of valid times
-     'hdr_typ':     nympy_integer_array,    # index of message type
-     'hdr_sid':     nympy_integer_array,    # index of station ID
-     'hdr_vld':     nympy_integer_array,    # index of valid time
-     'hdr_lat':     nympy_float_array,      # latitude
-     'hdr_lon':     nympy_float_array,      # longitude
-     'hdr_elv':     nympy_float_array,      # station elevation
-     'hdr_typ_table':   string_value,       # message types
-     'hdr_sid_table':   string_value,       # station IDs
-     'hdr_vld_table':   string_value,       # valid times "yyyymmdd_hhmmss"
-     'hdr_prpt_typ':    nympy_integer_array,   # optional
-     'hdr_irpt_typ':    nympy_integer_array,   # optional
-     'hdr_inst_typ':    nympy_integer_array,   # optional
-
-     # Observation data
-     'nobs':       integer_value,       # number of observation
-     'nobs_qty':   integer_value        # number of quality marks
-     'nobs_var':   integer_value        # number of variable names
-     'obs_qty':    nympy_integer_array, # index of quality mark
-     'obs_hid':    nympy_integer_array, # index of header
-     'obs_vid':    nympy_integer_array, # index of veriable or GRIB code
-     'obs_lvl':    nympy_float_array,   # pressure level
-     'obs_hgt':    nympy_float_array,   # height of observation data
-     'obs_val'     nympy_float_array,   # observatin value
-     'obs_qty_table':  string_array,    # quality marks
-     'obs_var_table':  string_array,    # variable names
-  }
+                plot_point_obs \
+                "PYTHON_NUMPY=MET_BASE/python/read_ascii_point.py sample_ascii_obs.txt" \
+                output_image.ps
+
+Both of the above examples use the **read_ascii_point.py** sample script which is included with the MET code. It reads ASCII data in MET's 11-column point observation format and stores it in a Pandas DataFrame to be read by the MET tools using Python embedding for point data. The **read_ascii_point.py** sample script can be found in:
+
+• MET installation directory in *MET_BASE/python*.
+
+• `MET GitHub repository <https://github.com/dtcenter/MET>`_ in *met/scripts/python*.
+
+.. _pyembed-mpr-data:
 
 Python Embedding for MPR data
 =============================

docs/Users_Guide/reformat_point.rst

@@ -538,19 +538,6 @@ In this example, the ASCII2NC tool will reformat the input **sample_ascii_obs.tx
 
 .. _ascii2nc-pyembed:
 
-Python Embedding for Point Observations
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Here is an example of processing the same set of observations but using Python embedding instead:
-
-.. code-block:: none
-		
-		ascii2nc -format python \
-		"MET_BASE/python/read_ascii_point.py sample_ascii_obs.txt" \
-		sample_ascii_obs_python.nc
-
-Please refer to :numref:`Appendix F, Section %s <appendixF>` for more details about Python embedding in MET.
-
 ascii2nc configuration file
 ---------------------------
 
@@ -1081,7 +1068,7 @@ For the GOES-16 and GOES-17 data, the computing lat/long is time consuming. So t
 
 When processing GOES-16 data, the **-qc** option may also be used to specify the acceptable quality control flag values. The example above regrids the GOES-16 AOD values to NCEP Grid number 212 (which QC flags are high, medium, and low), writing to the output the maximum AOD value falling inside each grid box.
 
-Listed below is an example of processing the same set of observations but using python embedding instead:
+Listed below is an example of processing the same set of observations but using Python embedding instead:
 
 .. code-block:: none
 		
@@ -1090,11 +1077,8 @@ Listed below is an example of processing the same set of observations but using
 		G212 python_gridded_ascii_python.nc -config Point2GridConfig_edr \
 		-field 'name="200"; level="*"; valid_time="20130827_205959";' -method MAX -v 1
 
-The user should replace the python script with the customized python script for the custom point observation data. This is an example for the python embedding.
-
 Please refer to :numref:`Appendix F, Section %s <appendixF>` for more details about Python embedding in MET.
 
-
 point2grid output
 -----------------
 

internal/test_unit/xml/unit_python.xml

@@ -495,22 +495,21 @@
     </output>
   </test>
 
-  <test name="python_plot_point_obs_CONFIG_XXX">
+  <test name="python_plot_point_obs_with_point_data">
     <exec>&MET_BIN;/plot_point_obs</exec>
     <env>
       <pair><name>TO_GRID</name> <value>NONE</value></pair>
     </env>
     <param> \
-      &OUTPUT_DIR;/pb2nc/ndas.20120409.t12z.prepbufr.tm00.nc \
-      &OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps \
-      -point_obs &OUTPUT_DIR;/ascii2nc/trmm_2012040912_3hr.nc \
-      -plot_grid &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \
+      'PYTHON_NUMPY=&MET_BASE;/python/read_ascii_point.py &MET_DATA;/sample_obs/ascii/precip24_2010010112.ascii' \
+      &OUTPUT_DIR;/python/precip24_2010010112.ps \
       -config &CONFIG_DIR;/PlotPointObsConfig \
-      -title "NAM 2012040900 F12 vs NDAS 500mb RH and TRMM 3h > 0" \
+      -plot_grid &DATA_DIR_MODEL;/grib2/nam/nam_2012040900_F012.grib2 \
+      -title "Precip24 from python embedding" -gc 61 \
       -v 3
     </param>
     <output>
-      <ps>&OUTPUT_DIR;/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps</ps>
+      <ps>&OUTPUT_DIR;/python/precip24_2010010112.ps</ps>
     </output>
   </test>