Yaml parsing errors with esm4.yaml

[GFDL-Eric]

Describe the bug

I'm using the fre/canopy module on Gaea.

I'm attempting to create a checkout script with the following code:

fre make create-checkout -npc -y esm4.yaml -t "prod openmp" -p ncrc5.intel23

I get the following error:

yaml.composer.ComposerError: found undefined alias 'name'
  in "esm4.yaml", line 58, column 57

if I comment out that directories section to eliminate the name issue, I get this error:

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!join'
  in "esm4.yaml", line 18, column 5

The files are from this directory:
https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/make/tests/ESM4_example/

To Reproduce

Load the fre/canopy module on Gaea, navigate to that directory after git cloning, and run the aforementioned command.

Expected behavior

A checkout script is generated.

Additional context
Add any other context about the problem here.

[singhd789]

This is being addressed in PR #168. Once that is merged in, we can try
fre make create-checkout -y esm4.yaml -p ncrc5.intel23 -t prod-openmp -npc

It should combine yamls necessary for compilation (model, compile, and platform yaml), parse, and create a checkout script

[GFDL-Eric]

Hi @singhd789,

I see that PR #168 has been merged... should I give the fre make command you referenced above a try with the new code?

[singhd789]

Hi @GFDL-Eric,

Currently the fre-cli auto-deployment in the fre/canopy module might not be working but you can still give the fre make command a go if you want! In order to get the more up-to-date fre-cli commands that were merged in, you can create your own conda environment.

1. On gaea, I believe the process you can follow is

module use -a /usw/conda/modulefiles
module load miniforge
conda create -n fre-cli fre-cli -c noaa-gfdl -c conda-forge
conda activate fre-cli

It might take a bit to build but to check if the fre-cli commands work, you can do fre make --help and fre make tools should pop up. Then you can follow one of these quickstarts with the esm yamls:
bare-metal (on-prem) guide --> https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/make/README.md#bare-metal-build
container guide --> https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/make/README.md#container-build

(esm 4.2 yaml examples: https://github.com/NOAA-GFDL/fre-cli/tree/main/fre/make/tests/ESM4_example)

If you give it a try, let me know of any other issues you see, questions or any next steps you're thinking to give a try.

[ceblanton]

Dana beat me to it. Yes, give it a try.

The fre/canopy (soon to be fre/2024.01) module is auto-deployed as a mostly reliable crontab every day at 5am, that does this:

conda env remove --name fre-cli --yes

conda create --name fre-cli fre-cli -c noaa-gfdl

To convince yourself, see the log file that should have done it:

/home/fms/log.update-fre-cli

That "conda create" will use the latest conda package auto-deployed to the 'noaa-gfdl' channel, which this repo's CI builds and uploads each push to main. This has traditionally been more reliable than the crontab but may not work sometimes.

https://anaconda.org/NOAA-GFDL/fre-cli

You can always install it yourself with those commands or the ones Dana suggested.

[singhd789]

@GFDL-Eric Just FYI first, With the ESM 4.2 yamls Ryan created, there are just slight formatting updates that should be used/merged. There's a open mr now (#200)

[GFDL-Eric]

@ceblanton just to confirm that the auto-deployment is not available on Gaea, correct? I see fre-cli 0.1.6 there while I see fre-cli 2024.01 on analysis after doing module load fre/canopy on each

[ceblanton]

You're right- the auto-deployment on gaea is definitely not presently working. The problem is that the scrontab job runs out of memory and I didn't yet try the probable solution yet, which is to request more cores.

The gfdl autodeployer has been much more reliable since I simplified it to blow away the entire environment and recreate each time. Anything less (updating or uninstalling and reinstalling) fails sometimes due to conda conflicts.

I'll give the gaea auto-deployment a fresh try now, thank you for the nudge.

[GFDL-Eric]

I'm now able to generate a compile script! I haven't tested actual compilation, however, as there are several "variables" in the MOM compile that need to be adjusted. Specifically, the MOM6_GIT_FIX_TAG and OCEAN_BGC_GIT_TAG aren't able to be passed to the additionalInstructions entry. Is there a tool that can make these "fixes" before the compile script is generated... presumably after the combined yaml is generated? Right now I can do it manually but maybe we should come up with something? Or we can take the long hard road of fixing how MOM is compiled in these xmls... should I create a new issue for this?

[ceblanton]

Yes, a new issue makes sense to me as it's no longer a parsing problem is it? Plus, this issue is closed, though it's nice that github still notifies us.

PS. the gaea autodeployment of fre-cli updates to fre/canopy should be fixed now that you've mastered your own installation :)

[GFDL-Eric]

Sounds good. Forgot to add, I was using the auto-deployment on gaea today for this. Thanks for fixing that!