[develop] Use templates for METplus config files #683
Conversation
… for Hera and modify run_WE2E_tests.py so that this location is obtained in a platform-independent way.
…ation of staged forecast files; modify code to correctly set VX_FCST_INPUT_BASEDIR in WE2E experiments.
…o that it is exactly consistent with the one used in run_WE2E_tests.py.
…t each task has its own values for NNODES, PPN, MEM, WTIME, MAXTRIES.
…t the DTC team wants.
…uired by NCO); also rename vx task top-level sections in config_defaults.yaml so that they're all lower case (to match names of ex-scripts).
…ks; Add ensemble member name (if running ensemble vx) to the names of the vx tasks in the ROCOTO xml.
…t convention; fix bugs in the two ex-scripts exregional_run_met_pointstat_ens[mean|prob].sh that call METplus twice.
…output. Details below. In the ex-scripts for EnsembleStat: (1) generate a new variable FCST_INPUT_FN_TEMPLATE that contains a list of forecast input template files for each of the ensemble members and use this in the METplus configuration files, and (2) change the input base directory (INPUT_BASE) to use VX_FCST_INPUT_BASEDIR, which depending on whether forecasts are being run is set either to the location of the UPP output from the forecasts or to the staged forecast UPP output (this allows staged ensemble staged files to be used to perform ensemble vx). In config_defaults.yaml, introduce new template variables (FCST_SUBDIR_TEMPLATE, FCST_FN_TEMPLATE, and FCST_FN_METPROC_TEMPLATE) that are/will be used in forming the variable FCST_INPUT_FN_TEMPLATE in the EnsembleStat ex-scripts.
…leStat for point-obs, ensure that METplus is called only once and for the correct variable (instead of for both SFC and UPA).
… This should later be replaced with a python version.
… This should later be replaced with a python version.
…able the ensemble member name (if doing an ensemble forecast). This makes it easier in METviewer to identify curves corresponding to different members.
…_LOCAL_MODULE_FN and VX_LOCAL_MODULE_FN; fix spacing.
|
@MichaelLueken, I think this PR is ready. The only problem is that @gsketefian is on leave this week and there is a minor conflict with the hera.yaml machine file (change in default paths). Are you comfortable with me resolving this conflict and then proceeding with the merge? |
|
@JeffBeck-NOAA I certainly don't mind if you went ahead and corrected the conflict. The issue is that PRs that change the FV3LAM_wflow.xml file are on hold until @christinaholtNOAA's PR #676 has been merged. @christinaholtNOAA Since both this PR and @chan-hoo's PR #682 have been approved and automated tests have been run and passed, can we move forward with these two PRs? Unfortunately, it would mean needing to apply these two updates to your PR before your work can be merged. Thank you very much for your time and consideration. |
|
@MichaelLueken, please merge my PR #682 before the PR #676 if it passed the tests :) I am going to create the |
|
@JeffBeck-NOAA Speaking with @christinaholtNOAA, the final changes should be in place and we are hoping that the Jenkins tests will be submitted today on PR #676. So long as there are no failures with the Jenkins tests and manual testing on Jet, then her PR will be merged either tomorrow or first thing Friday. If there are issues, then I will be able to move forward with the changes in this PR. Thanks! |
|
@MichaelLueken, thanks. I've resolved the conflicts for this PR and merged the commit. |
* [develop] Adds a YAML interface for creating a Rocoto XML. (ufs-community#676) Refactors the creation of a Rocoto XML to use a very generic Jinja2 template that is flexible enough to meet the needs of various workflow configurations supported by SRW. Specifically, it allows for a completely arbitrary workflow to be created under SRW, which includes the addition of completely arbitrary tasks on top of the predefined ones. --------- Co-authored-by: Michael Kavulich <[email protected]> * [develop] Change the build log output file extension from log to txt (ufs-community#690) When pipeline files are archived to s3 bucket, retrieving the file via a browser attempts to render/display files of known extensions. A browser doesn't generally understand what to do with a .log extension (e.g. build.log). For ease of use in the CI Dashboard, which is a static HTML page, the s3 archived build log needs a .txt extension (e.g. build.txt). * Add "MET_TOOL" definitions to new XML definition YAMLs * Fix incorrect YAML if block in config_defaults, remove non-needed "USCORE_ENSMEM_NAME_OR_NULL" variable * - Convert new test "MET_ensemble_verification_only_vx" to new YAML format - Fix f-string for utils.py error message * Fixing more failures (still more to go) * More fixes, got stand-alone verification test to pass! - Fix copy-paste errors in parm/workflow yamls - Update corrected variables for new names in exscripts * Improvement for monitor jobs script: if in debug mode, print the number of tasks that succeeded and failed for failed experiments * Forgot to include VX_FCST_INPUT_DIR definition for MET_ensemble_verification_only_vx test * Correct script for task_run_MET_EnsembleStat_vx_APCP * Pull out CATE and ENSMEM_INDEX from default VX_FCST_INPUT_DIR. My naive attempt to simplify things was the root of all my problems! * Everything working! Just need to solve problem of non-existent metatask dependencies! * Fix last failing ensemble test, fundamental tests and all verification tests now pass on Hera! --------- Co-authored-by: Christina Holt <[email protected]> Co-authored-by: Bruce Kropp - Raytheon <[email protected]>
…to be launched only after the previous task has completed even when there is no dependency of the task on the previous one.
…eads in the netcdf versions of the NDAS obs.
…e dependency section of the PointStat tasks, empty out the dependencies on the post and get_obs tasks since those are not run for this test.
…en performing time-lagging vx.
…place it only ub tasks that need it; define SLASH_ENSMEM_SUBDIR_OR_NULL locally in a loop when we loop over members in EnsembleStat vx task.
…PointStat task on get_obs_... and do_post tasks.
…at are looped over ensemble members).
…umber from name of METplus config and task log files.
…METplus conf files (MLCAPE and P-type).
|
@MichaelLueken I updated this PR and reran (on Hera) the four verification tests: In addition, I ran the fundamental tests, which currently are: All tests were successful. I think this PR is ready for the Jenkins tests. |
|
@MichaelLueken I see the error message "This commit cannot be buiild". I'd like the see the logs, but clicking on "Details" doesn't take me there (browser hangs). Are the logs not public? |
|
@gsketefian Unfortunately, the Jenkins logs are no longer public due to security concerns. I'll look through the failures now, but the due to the upgrading of PrgEnv on Gaea this week, Gaea is having issues. I'll let you know if there are any failures that will need to be addressed before this work can be merged. |
|
On Cheyenne GNU, the On Hera GNU, both the On Hera Intel, the Since the failures are expected and the rerun of the one unexpected failure successfully passed, I will now move forward with merging this work. |
|
@MichaelLueken Great, thank you! |
DESCRIPTION OF CHANGES:
This PR converts the METplus configuration files into Jinja templates form which the config files needed by the vx tasks can be generated. These config files are placed in the experiment directory. This prevents accidental changes to the METplus config files when multiple experiments are running simultaneously. This PR also reduces the number of J-jobs and ex-scripts used by the vx tasks by combining ones that are similar. Details:
_APCPgt01hin its name.fill_jinja_template.pyscript to generate the METplus config files from the templates.GridStatandPointStattools for deterministic verification into one ex-script with the stringgridstat_or_pointstatin its name.OBTYPE,ACCUM_HH(if necessary), andMET_TOOL; also, removeFHR).MET_ensemble_verification_only_vx) that tests use of staged obs and forecast files to run ensemble verification.Type of change
TESTS CONDUCTED:
Ran the
fundamentaltests on Hera with intel (see below); all passed.DOCUMENTATION:
Documentation is needed but will be added once the whole set of vx PRs has been merged (see Issue #630).
ISSUE:
Resolves #382
CHECKLIST