Convert staging job to python and yaml

[KateFriedman-NOAA]

Description

This PR converts the staging job from shell to python and introduces the use of yaml.

Changes in this PR:

1. Rename scripts/exglobal_stage_ic.sh to scripts/exglobal_stage_ic.py.
2. Update jobs/JGLOBAL_STAGE_IC to use .py script extension. Move COM* variable declarations and member loop down into yaml and python respectively. Move GDATE/gPDY/gcyc settings up to JJOB from ex-script and replace with newer cycle variables (as done in forecast job).
3. Create parm/stage folder to hold newly created stage.yaml.j2, which both mimics forecast-only functionality in existing scripts/exglobal_stage_ic.sh and adds functionality for cycled mode.
4. Create ush/python/pygfs/task/stage.py to house staging job python functions for call from scripts/exglobal_stage_ic.py.
5. Remove stage_ic job rocoto dependencies from xml. Do not need and removes area of duplicate maintenance.
6. Add cycled staging jobs for gdas and enkf suites.
7. Rename model_data to model for issue Change model_data to model in COM templates #2686

There will now be distinct stage_ic jobs for each RUN: gdasstage_ic, gfsstage_ic, enkfgdasstage_ic, stage_ic (for gefs).

Related work was done to set up new symlink folder structure under supported platform ICSDIR folder for use by updated staging job.

Resolves #2475
Resolves #2650
Resolves #2686

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

Ran all of the CI tests (either just staging jobs or end-to-end) on WCOSS2. Ran some of the CI tests on Hera and Hercules.

Ran some control CI tests from develop and compared outputs against test CI from staging branch. Outputs reproduced between control and test.

[emcbot]

Experiment C96C48_ufs_hybatmDA_13dfad25 FAIL on Wcoss2 at 08/19/24 06:14:29 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2651/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_13dfad25/logs/2024022318/enkfgdasfcst_mem001.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2651/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_13dfad25/logs/2024022318/enkfgdasfcst_mem002.log

Follow link here to view the contents of the above file(s): (link)

[emcbot]

CI Update on Wcoss2 at 08/19/24 08:56:04 PM
============================================
Cloning and Building global-workflow PR: 2651
with PID: 91691 on host: clogin03

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Mon Aug 19 21:06:34 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 08/19/24 09:46:15 PM
Case setup: Completed for experiment C48_ATM_ab35ffce
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_ab35ffce
Case setup: Skipped for experiment C48_S2SWA_gefs_ab35ffce
Case setup: Completed for experiment C48_S2SW_ab35ffce
Case setup: Completed for experiment C96_atm3DVar_extended_ab35ffce
Case setup: Skipped for experiment C96_atm3DVar_ab35ffce
Case setup: Completed for experiment C96_atmaerosnowDA_ab35ffce
Case setup: Completed for experiment C96C48_hybatmDA_ab35ffce
Case setup: Completed for experiment C96C48_ufs_hybatmDA_ab35ffce

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_ab35ffce *** SUCCESS *** at 08/19/24 11:00:15 PM
Experiment C48_S2SW_ab35ffce *** SUCCESS *** at 08/19/24 11:14:17 PM
Experiment C96C48_hybatmDA_ab35ffce *** SUCCESS *** at 08/19/24 11:56:21 PM
Experiment C96_atmaerosnowDA_ab35ffce *** SUCCESS *** at 08/20/24 12:49:24 AM
Experiment C96C48_ufs_hybatmDA_ab35ffce *** SUCCESS *** at 08/20/24 01:21:14 AM
Experiment C96_atm3DVar_extended_ab35ffce *** SUCCESS *** at 08/20/24 08:42:33 AM

[DavidHuber-NOAA]

Looks good!

[emcbot]

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2651

Experiment C48_ATM_ab35ffce Completed 1 Cycles: *SUCCESS* at Tue Aug 20 10:48:23 CDT 2024
Experiment C96_atm3DVar_ab35ffce Completed 3 Cycles: *SUCCESS* at Tue Aug 20 11:55:15 CDT 2024
Experiment C96C48_hybatmDA_ab35ffce Completed 3 Cycles: *SUCCESS* at Tue Aug 20 11:55:17 CDT 2024
Experiment C48_S2SWA_gefs_ab35ffce Completed 1 Cycles: *SUCCESS* at Tue Aug 20 12:08:11 CDT 2024
Experiment C48_S2SW_ab35ffce Completed 1 Cycles: *SUCCESS* at Tue Aug 20 12:25:27 CDT 2024

[KateFriedman-NOAA]

@NeilBarton-NOAA This PR has just been merged into develop. Let me know how I can help update your PR branch to accommodate the staging job refactor.

[NeilBarton-NOAA]

Thanks @KateFriedman-NOAA, I'm currently testing, and I will let you know when I have questions.

[guillaumevernieres]

@KateFriedman-NOAA , It looks like this PR broke the WCDA/gfsv17 tests. Was that tested? I see the hera-ci green but I don't see what was actually tested.

Edit: just seeing @RussTreadon-NOAA 's issue ... I suspect that's my issue as well, so ignore for now