Feature 1844 python embedding

[hsoh-u]

Expected Differences

• Added met/src/libcode/vx_pointdata_python

• Added two python scripts to met/scripts/python (met_point_obs.py & read_met_point_obs.py: reaing MET point observation NetCDF by python)

• Redesign and implement the class structure at met/src/libcode/vx_nc_obs

  • Rename NcHeaderData to MetPointHeader
  • Moved many APIs from NcPointObsData to MetPointObsData (base class, obs data only))
  • Moved many APIs from MetNcPointObs to MetPointData (header & obs)

• Added pyobject_as_bool to met/src/libcode/vx_python3_utils

• Support python embedding for 4 tools

• Added 4 unittests (NetCDF input is changed to python embedding, so the same output should be generated)

• Added documentations

• Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
  
  If yes, please describe:
  

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [No]
  
  If yes, please describe:
  

Pull Request Testing

• Describe testing already performed for these changes:
  

  • Run unittest and make sure the outputs are not changed
  • Run unittest and compare new outputs (NetCDF input & python embedding)
    • <unit_test_output>/python/pb2nc_TMP.nc VS. <unit_test_output>/point2grid/pb2nc_TMP.nc
    • <unit_test_output>/python/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps VS. <unit_test_output>/plot_point_obs/nam_and_ndas.20120409.t12z.prepbufr_CONFIG.ps
    • <unit_test_output>/python/point_stat_GRIB1_NAM_GDAS_WINDS_120000L_20120409_120000V.stat VS. <unit_test_output>/python/point_stat_GRIB1_NAM_GDAS_WINDS_120000L_20120409_120000V.stat
    • <unit_test_output>/python/ensemble_stat VS. <unit_test_output>/ensemble_stat/

The outputs are available at /d1/personal/hsoh/git/pull_request/MET_feature_1844_python_embedding/unit_test_output/

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

  • run a full unit test
  • Just change <netcdf_input> to ''PYTHON_NUMPY=MET_BASE/python/read_met_point_obs.py <netcdf_input>' for 4 tools

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

  • More outputs by new unit tests

• Please complete this pull request review by [Fill in date].
  

Pull Request Checklist

See the METplus Workflow for details.

• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Linked issues with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

Running a full regression test in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta5/feature_1844 reveals unexpected differences. Specifically in the files listed below:

egrep -i "ERROR:|file1:" test_regression.log | egrep -B 2 ERROR: | grep ensemble | grep file1
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_ecnt.txt
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.nc
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V_orank.txt
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_NCEP_1.0DEG_20120410_120000V.stat
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ecnt.txt
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V_ens.nc
file1: MET-feature_1844_python_embedding/test_output/climatology/ensemble_stat_ONE_CDF_BIN_20120410_120000V.stat

These differences require further investigation. No explanation for why they should differ is listed in the pull request.

Since we're otherwise ready to do the beta5 release, I recommend we do the following:

• Proceed with the beta5 release on 1/14/22.
• Move issue MET#1884 and this PR MET#2010 over to the beta6 project.
• After completing the beta5 release, merge the latest from develop into this feature branch. Also, update the develop-ref version to get past of lots of existing diffs in the output.
• Rerun the full regression test to compare the update version of this feature branch to the newly updated develop-ref.
• That should allow us to see the real diffs to investigate rather than wasting time sifting through lots of other diffs.

[JohnHalleyGotway]

After the met-10.1.0-beta5 release, I updated develop-ref to get past diffs from many recent changes. On 1/18/22, I merged the latest from develop into this feature branch, feature_1844_python_embedding, and initiated a regression test on kiowa in:
/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta6/feature_1844

Once the regression test completes, I'll post the list of diffs here.

[hsoh-u]

I merged the latest develop branch into the feature branch and have the same difference.s I will take a look

OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_ENS CRPS CRPSS IGN ME RMSE SPREAD ME_OERR RMSE_OERR SPREAD_OER
OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_ENS CRPS CRPSS IGN ME RMSE SPREAD ME_OERR RMSE_OERR SPREAD_OE
NA NA NA ECNT 361 5 2.15778 0.23127 42.96489 0.024977 3.16292 1.03873 NA NA
NA NA NA ECNT 253 5 2.22308 0.2312 39.93187 0.11707 3.22139 1.03314 NA NA

[hsoh-u]

The differences at ensemble_stat was fixed and I got the same result.

[JohnHalleyGotway]

Sorry for the delay on this @hsoh-u. I'm going to merge develop into this feature branch and rerun a regression test to confirm. Thanks!

[JohnHalleyGotway]

I approve of these changes.

I ran a full regression test vs develop in kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta6/feature_1844 and confirmed that only diffs are the addition of new output files. I skimmed through the code changes and didn't see any obvious problems. Thanks for adding more tests to unit_python.xml to demonstrate this functionality!