CMORizing reanalysis data

[glpotter]

I need to CMORize several sets of monthly reanalysis data. I would very much like to use esmvaltool but I'm having trouble setting up the recipe. Several of the example recipes don't exist. For example:

(esmvaltool) potter@linux:~/esmvaltool_tutorial/esmvaltool_recipes$ esmvaltool recipes get cmorizers/cmorizers/recipe_era5-land.yml
2022-06-13 22:19:01,492 UTC [157218] INFO Copying installed recipe to the current folder...
2022-06-13 22:19:01,492 UTC [157218] ERROR Program terminated abnormally, see stack trace below

For example, I need to cmorize monthly average MERRA2, JRA55, and CFSR (all CMIP5 variables for 1980-2021) . Do you have an example of how I might set this up?

Thank you very much.

[zklaus]

Support for most observational datasets in ESMValTool does not come via recipes, but rather via a special cmorization facility for observations. This is described in the documentation here. For example for merra2, you should be able to do something like

esmvaltool data prepare --config_file <config-user.yml> MERRA2

to both download and reformat the observations.

You may find that that does not include all variables that you are interested in. In that case, it may be necessary to modify the configuration at esmvaltool/cmorizers/data/cmor_config/MERRA2.yml, but I would suggest to start with the configuration as present in the release 2.5.0 of ESMValTool and to move to extensions from there.

I'm happy to continue the discussion here and to see how you are getting on.

[glpotter]

I was going to try to modify the input for my purpose but before that, I tried your suggestion:
obviously, I'm doing something wrong. I have the .yml in my home directory.
Here is what I get:

(esmvaltool) potter@linux:~$ esmvaltool data prepare --config_file MERRA2.yml MERRA2
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 499, in run
fire.Fire(ESMValTool())
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 681, in CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/cmorizers/data/cmorizer.py", line 521, in prepare
self.formatter.start('preparation', datasets, config_file, kwargs)
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvaltool/cmorizers/data/cmorizer.py", line 58, in start
self.config = read_config_user_file(config_file, f'data{command}',
File "/home/potter/mambaforge/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_config/_config.py", line 147, in read_config_user_file
if 'obs4mips' in cfg[section]:
KeyError: 'rootpath'

Thank you!

[bouweandela]

It looks like you may be mixing up the different types of configuration file. The --config_file argument takes a user configuration file.

[glpotter]

I hopelessly confused on how to cmorize a reanalysis data set. Your instructions are very clear to someone more familiar with the processing. I would like very much to use your tools but I am afraid my limited understanding of how esmvaltool works is a problem. Previously, we were using CMOR plus a processor that used resource files similar to your config files. The person who wrote the processor has departed and the code no longer works on the latest operating systems. We must modernize the processing and it looks like esmvaltool is the answer. Our project is a NASA service called CREATE (collaborative reanalysis technical environment) and is a replacement for ana4mips. We have cmorized all of the major reanalyses from 1979-2019 both monthly and 6 hour but now I am retired and we need to pass the project on. I would like to refer our users to esmvaltool so that if they are interested in extending the CREATE service they could use your tools. Here is a reference to our service: DOI: https://doi.org/10.1175/BAMS-D-17-0174.1

[bouweandela]

Would it work for you to have a video call so I can guide you through the process? You can contact me by sending an email to [email protected]. We regularly get the question from users if we can share CMORized data, so your project looks extremely useful to the users of our tool.

[glpotter]

Yes, this would be great. I have emailed you with possible time.

[bouweandela]

It looks like there is a typo in the command you're running, it should be

esmvaltool recipes get cmorizers/recipe_era5-land.yml

You can view all available recipes with the command

esmvaltool recipes list

But like @zklaus says, for cmorizing there is the esmvaltool data command. Note that some datasets are supported without prior CMORization, such as ERA5. The recipes in the cmorizers directory can indeed be used to cmorize ERA5 data, but this is not needed for using the data with the ESMValTool.

[glpotter]

Thanks, It works now

[glpotter]

I have modified the MERRA2.yml so that it contains nearly all of the CMIP5 standard 2D variables. The remaining variables are combination of variables that are found in the "rad" monthly collection. The merra-2.py recipe will need to be modified. This is a list of the differences that need to be calculated. Can this be done in the merra2.py recipe?

rsus_monthly:
short_name: rsus
mip: Amon
raw: SWGDN-SWGNT
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsuscs_monthly:
short_name: rsuscs
mip: Amon
raw: SWGDNCLR-SWGNTCLR
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsut_monthly:
short_name: rsut
mip: Amon
raw: SWTDN-SWTNT
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsutcs_monthly:
short_name: rsutcs
mip: Amon
raw: "SWTDN-SWTNTCLR"
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'

Contents of MERRA2.yml below:
--`-

Global attributes of NetCDF file

attributes:
dataset_id: MERRA2
project_id: OBS6
tier: 3
version: '5.12.4'
modeling_realm: reanaly
source: ''
reference: 'merra2'
comment: |
'Contains modified MERRA-2 data'

Variables to CMORize

variables:

monthly frequency

2 dimensional variables

clt_monthly:
short_name: clt
mip: Amon
raw: CLDTOT
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
pr_monthly:
short_name: pr
mip: Amon
raw: PRECTOT
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
evspsbl_monthly:
short_name: evspsbl
mip: Amon
raw: EVAP
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
hfss_monthly:
short_name: hfss
mip: Amon
raw: HFLUX
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
hfls_monthly:
short_name: hfls
mip: Amon
raw: EFLUX
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
huss_monthly:
short_name: huss
mip: Amon
raw: QV2M
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
prc_monthly:
short_name: prc
mip: Amon
raw: PRECCON
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
prsn_monthly:
short_name: prsn
mip: Amon
raw: PRECSNO
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
prw_monthly:
short_name: prw
mip: Amon
raw: TQV
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
ps_monthly:
short_name: ps
mip: Amon
raw: PS
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
psl_monthly:
short_name: psl
mip: Amon
raw: SLP
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
rlds_monthly:
short_name: rlds
mip: Amon
raw: LWGAB
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rldscs_monthly:
short_name: rlds
mip: Amon
raw: LWGABCLR
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rlus_monthly:
short_name: rlus
mip: Amon
raw: LWGEM
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rlut_monthly:
short_name: rlut
mip: Amon
raw: LWTUP
file: 'MERRA2_?00.tavgM_2d_rad_Nx.{year}??.nc4'
rlutcs_monthly:
short_name: rlutcs
mip: Amon
raw: LWTUPCLR
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsds_monthly:
short_name: rsds
mip: Amon
raw: SWGDN
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsdscs_monthly:
short_name: rsdscs
mip: Amon
raw: SWGDNCLR
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
rsdt_monthly:
short_name: rsdt
mip: Amon
raw: SWTDN
file: 'MERRA2_???.tavgM_2d_rad_Nx.{year}??.nc4'
tas_monthly:
short_name: tas
mip: Amon
raw: T2M
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
tasmax_monthly:
short_name: tasmax
mip: Amon
raw: T2MMAX
file: 'MERRA2_???.statM_2d_slv_Nx.{year}??.nc4'
tasmin_monthly:
short_name: tasmin
mip: Amon
raw: T2MMIN
file: 'MERRA2_???.statM_2d_slv_Nx.{year}??.nc4'
tauu_monthly:
short_name: tauu
mip: Amon
raw: TAUX
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
tauv_monthly:
short_name: tauv
mip: Amon
raw: TAUY
file: 'MERRA2_???.tavgM_2d_flx_Nx.{year}??.nc4'
ts_monthly:
short_name: ts
mip: Amon
raw: TS
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
uas_monthly:
short_name: uas
mip: Amon
raw: U10M
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'
vas_monthly:
short_name: vas
mip: Amon
raw: V10M
file: 'MERRA2_???.tavgM_2d_slv_Nx.{year}??.nc4'

[bouweandela]

Friendly ping @valeriupredoi