-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Simplify the way the configuration of the vx is handled #1082
[develop] Simplify the way the configuration of the vx is handled #1082
Conversation
… vx configuraion file.
…recast and observation values) deterministic vx yaml config file directly into ex-script for deterministic GridStat and PointStat tasks to form an overall yaml file from which to generate METplus config files from template.
…e for deterministic vx.
…recast and observation values) ensemble vx yaml config file directly into ex-script for (ensemble) GenEnsProd tasks to form an overall yaml file from which to generate METplus config files from template.
… purely yaml ensemble vx configuration file.
…th forecast and observation values) ensemble vx yaml config file directly into ex-script for GridStat ensemble mean tasks to form an overall yaml file from which to generate METplus config files from template.
…th forecast and observation values) ensemble vx yaml config file directly into ex-script for GridStat ensemble probabilistic tasks to form an overall yaml file from which to generate METplus config files from template.
…th forecast and observation values) ensemble vx yaml config file directly into the ex-script for PointStat ensemble mean tasks to form an overall yaml file from which to generate METplus config files from the PointStat_ensmean.conf template.
…th forecast and observation values) ensemble vx yaml config file directly into the ex-script for PointStat ensemble probabilistic tasks to form an overall yaml file from which to generate METplus config files from the PointStat_ensprob.conf template.
…nce the parsing (of items in the vx configuration yaml files, where items can be field names, levels, and thresholds, to obtain the forecast and observation values) is now done within the METplus config file templates (i.e. it's done by Jinja).
@gsketefian I ran tests on Hera. It failed for some tests (that passed for you before): /scratch2/NCEPDEV/fv3-cam/Ratko.Vasic/SRW-1082/expt_dirs/MET_ensemble_verification_only_vx_time_lag/log/get_obs_ndas_2021050500.log Can you please take a look, in second log fie, it is trying to get from HPSS file that does not exist. |
I saw similar issues with the |
…alues in a single location in the jinja macros file instead of in each METplus config template.
@MichaelLueken @RatkoVasic-NOAA I just pushed a few more commits that further streamline the METplus config templates. I reran the test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The verification WE2E tests successfully ran through to completion on Hera:
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
MET_ensemble_verification_only_vx_20240510132442 COMPLETE 1.32
MET_ensemble_verification_only_vx_time_lag_20240510132445 COMPLETE 3.93
MET_ensemble_verification_winter_wx_20240510132448 COMPLETE 107.94
MET_verification_20240510132449 COMPLETE 10.91
MET_verification_only_vx_20240510132451 COMPLETE 0.23
MET_verification_winter_wx_20240510132452 COMPLETE 17.66
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202405 COMPLETE 44.38
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024051013245 COMPLETE 20.15
----------------------------------------------------------------------------------------------------
Total COMPLETE 206.52
Approving PR now.
After two tries, it passed on Hera:
|
@gsketefian , can you please check what is going on with this one:
It is failing randomly. Yesterday it failed twice in a row causing rocoto to give up and whole test to fail.
I wonder if you have an idea what might be causing this random occurrence? |
@RatkoVasic-NOAA I think I figured out the problem. What's happening is that during the There are a couple of quick-and-dirty (i.e. hard-coded) fixes for this, but I'd rather implement a more thorough fix. Since this is something that predates this PR and is not directly related, I'd like to create an issue and take a look next week. Are you ok with that? Otherwise I can do a one-liner quick fix. |
@gsketefian great! I suggest you do 'dirty' quick fix now (although not related to this PR), and open an issue for nice fix for later. @MichaelLueken what do you think? I'll approve this PR either way. |
@RatkoVasic-NOAA I'd recommend opening an issue to properly address this issue rather than create a temporary fix. A quick-and-dirty fix could potentially break the way the workflow runs for certain tests, leading to more issues. I'd err on the side of caution and focus on a fix that will work for all verification tests. @gsketefian Once the |
…nkins pipeline when running the srw_metric.sh script).
@MichaelLueken @RatkoVasic-NOAA @EdwardSnyder-NOAA I added back the file |
Except expected failures which are reported in issue #1084, tests passed on Hera:
Approved. |
The Jenkins tests successfully passed on all platforms. Moving forward with merging this work now. |
DESCRIPTION OF CHANGES:
This PR removes the tasks
parse_vx_config_[det|ens]
that read in and parse the verification (vx) yaml configuration files (one for deterministic vx and another for ensemble vx). Currently, these tasks are used to call the scriptdecouple_fcst_obs_vx_config.py
to separate the "coupled" vx config information in the yaml files into forecast and observation components. For example, a "coupled" field name of "CRAIN%%PRWE" in the deterministic configuration file corresponds to a field name of "CRAIN" for forecasts and "PRWE" for the observations (and the "%%" is the delimiter). After separating each item (field name, level, or threshold) in this way, theparse_vx_config_[det|ens]
tasks write this separated information into two new intermediate configuration files (one for deterministic and one for ensemble) as two separate dictionaries (one for forecasts and another for observations). These intermediate config files are then read in by the various vx task ex-scripts.In this PR, the
parse_vx_config_[det|ens]
tasks and thedecouple_fcst_obs_vx_config.py
script are removed (so that the intermediate configuration files are no longer created). The separation into forecast and observation values of the "coupled" information in the vx configuration files is now performed in the jinja2 templates for the METplus configuration files, hiding these details from the user.This PR was prompted by some of the questions by @mkavulich in PR #1005.
Type of change
TESTS CONDUCTED:
The following vx WE2E tests were run on Hera:
They all completed successfully.
DOCUMENTATION:
No updates necessary.
ISSUE:
None.
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):
@mkavulich