Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change model_data to model in COM templates #2686

Closed
2 tasks
WalterKolczynski-NOAA opened this issue Jun 13, 2024 · 14 comments · Fixed by #2651
Closed
2 tasks

Change model_data to model in COM templates #2686

WalterKolczynski-NOAA opened this issue Jun 13, 2024 · 14 comments · Fixed by #2651
Assignees
Labels
feature New feature or request

Comments

@WalterKolczynski-NOAA
Copy link
Contributor

What new functionality do you need?

By NCO's request, model_data should be just model in COM paths.

What are the requirements for the new functionality?

model_data replaced by model.

Acceptance Criteria

  • Output that previous landed in model_data now lands in model
  • Output is otherwise unchanged

Suggest a solution (optional)

No response

@KateFriedman-NOAA
Copy link
Member

@RussTreadon-NOAA @CoryMartin-NOAA FYI about this request from NCO to change model_data subfolder name to just model. I see the following instances of model_data within the gsi_enkf.fd and gdas.cd subcomponents that should be updated to be consistent. I am taking care of the update on the workflow side.

sorc/gsi_enkf.fd/regression/global_enkf.sh:     $nln $datens/$memchar/model_data/atmos/history/${prefix_ens}.atmf00${fhr}.nc sfg_${global_adate}_fhr0${fhr}_${memchar}
sorc/gsi_enkf.fd/regression/global_enkf.sh:            $nln $datens/$memchar/model_data/atmos/history/${prefix_ens}sfcf00${fhr}.nc sfgsfc_${global_adate}_fhr0${fhr}_${memchar}
sorc/gsi_enkf.fd/regression/global_enkf.sh:    $nln $datens/ensstat/model_data/atmos/history/${prefix_ens}.atmf00${fhr}.ensmean.nc sfg_${global_adate}_fhr0${fhr}_ensmean
sorc/gsi_enkf.fd/regression/global_4denvar.sh:dathis=$COMROOTgfs/$dumpges.$PDYg/${cycg}/model_data/atmos/history
sorc/gsi_enkf.fd/regression/global_4denvar.sh:$nln $datens/ensstat/model_data/atmos/history/${prefix_ens}.sfcf006.ensmean.nc         ./sfcf06_anlgrid
sorc/gsi_enkf.fd/regression/global_4denvar.sh:  $nln $datens/$member/model_data/atmos/history/$sigens ${ENS_PATH}sigf${fh}_ens_${member}
sorc/gdas.cd/ush/soca/run_jjobs.py:        model_data = os.path.join(self.com_src, f"{self.RUN}.{self.gPDY}", self.gcyc, "model_data")
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ocean_history_src = os.path.join(model_data, 'ocean', 'history')
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ice_history_src = os.path.join(model_data, 'ice', 'history')
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ice_restart_src = os.path.join(model_data, 'ice', 'restart')
sorc/gdas.cd/ush/soca/marine_recenter.py:                                       'model_data',
sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py:                                  'model_data', domain, 'history')
sorc/gdas.cd/test/atm/global-workflow/jjob_ens_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_ens_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos/history
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos/history
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos/history
sorc/gdas.cd/test/snow/apply_jedi_incr.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/create_bkg_ens.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/test_imsproc.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/letkfoi_snowda.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/aero/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ice/history
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ice/restart
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ocean/history
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/atmos/analysis
sorc/gdas.cd/test/soca/gw/prep.sh:cp ${ice_rst} ${COM}/06/model_data/ice/restart/20180415.120000.cice_model.res.nc
sorc/gdas.cd/test/soca/gw/prep.sh:  cp $ocnf ${COM}/06/model_data/ocean/history/gdas.ocean.t06z.inst.f00$i.nc
sorc/gdas.cd/test/soca/gw/prep.sh:  cp $hist_icef ${COM}/06/model_data/ice/history/gdas.ice.t06z.inst.f00$i.nc
sorc/gdas.cd/test/soca/gw/prep.sh:    cp ${COM}/06/model_data/ocean/history/gdas.ocean.t06z.inst.f003.nc \
sorc/gdas.cd/test/soca/gw/prep.sh:    list_of_ocn_fcst=$(ls ${COM}/06/model_data/${domain}/history/gdas.${domain}.t06z.inst.f*.nc)
sorc/gdas.cd/test/soca/gw/prep.sh:    oceandir=${COMENS}/06/mem00${mem}/model_data/ocean/history
sorc/gdas.cd/test/soca/gw/prep.sh:    icedir=${COMENS}/06/mem00${mem}/model_data/ice/history

@KateFriedman-NOAA
Copy link
Member

@GeorgeGayno-NOAA FYI about this request from NCO to change the model_data subfolder name to just model. I see the following instance of model_data within the UFS_UTILS subcomponent. Please update to be consistent, thanks!

sorc/ufs_utils.fd/util/gdas_init/copy_coldstart_files.sh:  SAVEDIR_MODEL_DATA=$SUBDIR/model_data/atmos/input

@RussTreadon-NOAA
Copy link
Contributor

@KateFriedman-NOAA , thank you for letting us know about this NCO requirement. As you note, the g-w changes need to be coordinated with GSI and GDASApp changes. Would you please open a GSI issue and a GDASApp issue to document the required change?

@KateFriedman-NOAA
Copy link
Member

@RussTreadon-NOAA As requested, issues opened: NOAA-EMC/GSI#761 & NOAA-EMC/GDASApp#1198

@GeorgeGayno-NOAA
Copy link
Contributor

@GeorgeGayno-NOAA FYI about this request from NCO to change the model_data subfolder name to just model. I see the following instance of model_data within the UFS_UTILS subcomponent. Please update to be consistent, thanks!

sorc/ufs_utils.fd/util/gdas_init/copy_coldstart_files.sh:  SAVEDIR_MODEL_DATA=$SUBDIR/model_data/atmos/input

Will work this under: ufs-community/UFS_UTILS#967

@RussTreadon-NOAA
Copy link
Contributor

NOAA-EMC/GSI has been updated to use model instead of model_data. Done at 6a87460. See GSI PR #762 for details.

@RussTreadon-NOAA
Copy link
Contributor

NOAA-EMC/GDASApp:feature/model has been updated to use model instead of model_data. Build and run GDASApp ctests on Orion inside working copy of PR #2700 g-w. All 48 tests pass but notice that model_data remains. This makes sense sinc PR #2700 g-w still uses model_data. See GDASApp issue #1198 for additional information.

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

@KateFriedman-NOAA
Copy link
Member

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

Hmmmmm, I am considering options for this, one being to merge this work with my issue #2475 work (since I need this work for that issue's work and merging the efforts would reduce work for both of them).

Question for @RussTreadon-NOAA and @GeorgeGayno-NOAA ...are the model_data -> model changes going into your respective develop branches ahead of Orion Rocky9 updates?

@RussTreadon-NOAA
Copy link
Contributor

@KateFriedman-NOAA ,

GDASApp

  • Orion Rocky 9 is already in GDASApp develop. Done via PR #1180
  • model_data -> model change remains feature/model. PR not opened since change needs to be coordinated with updates to GDASApp test case data directories on WCOSS2, Hera, and Orion/Hercules

GSI

  • Orion Rocky 9 on pause (see GSI PR #764) given the increased gsi.x and enkf.x wall times
  • model_data -> model change is already in GSI develop. Done via PR #762

@KateFriedman-NOAA
Copy link
Member

@RussTreadon-NOAA Ok, given the mixture of order between the model_data -> model changes and Orion Rocky9 updates between the subcomponents I will keep this work separate from the issue #2475 work. I will need to create symlinks within the current IC structure to support the model_data -> model changes going in before my issue #2475 work (which also handles that folder name change but which I don't want to mix with the Orion Rocky9 updates). I'll create the symlinks and create a g-w branch with the workflow side changes. Stay tuned. Thanks!

@GeorgeGayno-NOAA
Copy link
Contributor

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

Hmmmmm, I am considering options for this, one being to merge this work with my issue #2475 work (since I need this work for that issue's work and merging the efforts would reduce work for both of them).

Question for @RussTreadon-NOAA and @GeorgeGayno-NOAA ...are the model_data -> model changes going into your respective develop branches ahead of Orion Rocky9 updates?

The directory changes were merged yesterday: ufs-community/UFS_UTILS#968

The Orion changes were merged this morning: ufs-community/UFS_UTILS#966

@KateFriedman-NOAA
Copy link
Member

Noted, thanks @GeorgeGayno-NOAA !

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 26, 2024
@AndrewEichmann-NOAA
Copy link
Contributor

Is anybody working on making this change in the template definitions?

@KateFriedman-NOAA
Copy link
Member

Is anybody working on making this change in the template definitions?

If you're referring to the COM templates in config.com, then yes, I have the updates in a fork branch of mine but have been testing them as part of work for issue #2475. May combine those updates.

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 22, 2024
WalterKolczynski-NOAA added a commit that referenced this issue Aug 20, 2024
This PR converts the staging job from shell to python and introduces the
use of yaml.

Changes in this PR:

1. Rename `scripts/exglobal_stage_ic.sh` to
`scripts/exglobal_stage_ic.py`.
2. Update `jobs/JGLOBAL_STAGE_IC` to use `.py` script extension. Move
`COM*` variable declarations and member loop down into yaml and python
respectively. Move `GDATE/gPDY/gcyc` settings up to JJOB from ex-script
and replace with newer cycle variables (as done in forecast job).
3. Create `parm/stage` folder to hold newly created `stage.yaml.j2`,
which both mimics forecast-only functionality in existing
`scripts/exglobal_stage_ic.sh` and adds functionality for cycled mode.
4. Create `ush/python/pygfs/task/stage.py` to house staging job python
functions for call from `scripts/exglobal_stage_ic.py`.
5. Remove `stage_ic` job rocoto dependencies from xml. Do not need and
removes area of duplicate maintenance.
6. Add cycled staging jobs for gdas and enkf suites.
7. Rename `model_data` to `model` for issue #2686

There will now be distinct `stage_ic` jobs for each `RUN`:
`gdasstage_ic`, `gfsstage_ic`, `enkfgdasstage_ic`, `stage_ic` (for
gefs).

Related work was done to set up new symlink folder structure under
supported platform `ICSDIR` folder for use by updated staging job.

Resolves #2475
Resolves #2650
Resolves #2686

---------

Co-authored-by: Rahul Mahajan <[email protected]>
Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
Co-authored-by: David Huber <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants