Added EMAC CMORizer

[schlunma]

Description

This PR adds a CMORizer for native EMAC model output.

Changes not directly linked to EMAC

• Added possibility to filter warnings for the load preprocessor step in the config-developer.yml file, e.g.,

ignore_warnings:
    load:
      - {message: 'Missing CF-netCDF formula term variable .*, referenced by netCDF variable .*', module: iris}

ESMValCore/esmvalcore/_recipe.py

Lines 727 to 734 in ccb53b8

	def _update_warning_settings(settings, project):
	"""Update project-specific warning settings."""
	cfg = get_project_config(project)
	if 'ignore_warnings' not in cfg:
	return
	for (step, ignored_warnings) in cfg['ignore_warnings'].items():
	if step in settings:
	settings[step]['ignore_warnings'] = ignored_warnings

and

ESMValCore/esmvalcore/preprocessor/_io.py

Lines 113 to 138 in ccb53b8

	def load(file, callback=None, ignore_warnings=None):
	"""Load iris cubes from files."""
	logger.debug("Loading:\n%s", file)
	if ignore_warnings is None:
	ignore_warnings = []
	ignore_warnings.append({
	'message': "Missing CF-netCDF measure variable .*",
	'category': UserWarning,
	'module': 'iris',
	})
	ignore_warnings.append({
	'message': "Ignoring netCDF variable '.*' invalid units '.*'",
	'category': UserWarning,
	'module': 'iris',
	})
	with catch_warnings():
	for warning_kwargs in ignore_warnings:
	warning_kwargs.setdefault('action', 'ignore')
	filterwarnings(**warning_kwargs)
	raw_cubes = iris.load_raw(file, callback=callback)
	logger.debug("Done with loading %s", file)
	if not raw_cubes:
	raise Exception('Can not load cubes from {0}'.format(file))
	for cube in raw_cubes:
	cube.attributes['source_file'] = file
	return raw_cubes

---

Closes #312
Closes #309
Closes #139
Closes #66

Related PRs:

• Added custom CMOR tables used for EMAC CMORizer #1599
• Added derivation of hfns #1594
• Removed trailing whitespace in custom CMOR tables #1564

Link to documentation:

• https://esmvaltool--1554.org.readthedocs.build/projects/ESMValCore/en/1554/quickstart/configure.html#filter-preprocessor-warnings
• https://esmvaltool--1554.org.readthedocs.build/projects/ESMValCore/en/1554/quickstart/find_data.html#emac

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

[codecov]

Codecov Report

Merging #1554 (d41fb20) into main (db7ca97) will increase coverage by 0.25%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1554      +/-   ##
==========================================
+ Coverage   90.98%   91.23%   +0.25%     
==========================================
  Files         201      203       +2     
  Lines       10680    10978     +298     
==========================================
+ Hits         9717    10016     +299     
+ Misses        963      962       -1     

Impacted Files	Coverage Δ
esmvalcore/_recipe.py	95.86% <100.00%> (+0.03%)	⬆️
esmvalcore/cmor/_fixes/emac/_base_fixes.py	100.00% <100.00%> (ø)
esmvalcore/cmor/_fixes/emac/emac.py	100.00% <100.00%> (ø)
esmvalcore/cmor/_fixes/shared.py	100.00% <100.00%> (ø)
esmvalcore/preprocessor/_io.py	87.39% <100.00%> (+0.84%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db7ca97...d41fb20. Read the comment docs.

[valeriupredoi]

what's worse - the tests/integration/cmor/_fixes/cmip6/test_cesm2.py test I just modified gets skipped - no reason to be skipped, even though there is a very ancient skip decorator attached to it?

[valeriupredoi]

scratch that! am a dummer - it's sequential

[zklaus]

Not suggesting that you must do this change here and now, but mid- to long-term we should replace all of these __file__ references with importlib.resources.

[valeriupredoi]

aye, @bouweandela is right - no sign of the emac.nc file in the (correct) test data root:

• TEST DATA PATH CESM2 /root/project/tests/integration/cmor/_fixes/test_data
• dir /root/project/tests/integration/cmor/_fixes/test_data contains 'cesm2_cl.nc', 'cesm2_waccm_cl.nc', 'cnrm_cm6_1_cl.nc', 'common_cl_a.nc', 'common_cl_ap.nc', 'common_cl_hybrid_height.nc', 'create_test_data.py', 'gfdl_cm4_cl.nc', 'icon_2d.nc', 'icon_3d.nc', 'icon_grid.nc'

[bouweandela]

The problem can indeed be fixed by installing git and ssh before checkout on CircleCI. See #1601 for more info.

[axel-lauer]

I looked through the plots produced by the EMAC example recipe recipe_emac_new.yml and most things look good - as far as I can tell. There are, however, a few items that might need to be double-checked:

• map of precip (given in kg/m2/s): the color scale ranges from 0-10, i.e. the plot is empty and evaluation is not possible.
• map of zg at 500 hPa: the values over Antarctica and over Greenland look very strange, I would expect the values to range somewhere between 5000 and 6000 m but not between 0 and 3000 m as in the plot.
• time series of surface pressure: assuming that the values shown are global mean values, the absolute values and the variability look wrong

[schlunma]

Thanks Axel for having a look at this!!

* map of precip (given in kg/m2/s): the color scale ranges from 0-10, i.e. the plot is empty and evaluation is not possible.

Ah, yes, apparently the monitoring diagnostics expects units of mm day-1. Unfortunately, it is still not possible to do this conversion in the preprocessor (#1574). Here is the output of ncview, I think the values makes sense:

* map of zg at 500 hPa: the values over Antarctica and over Greenland look very strange, I would expect the values to range somewhere between 5000 and 6000 m but not between 0 and 3000 m as in the plot.

That's also present in the raw data (see below). Note that the CMORizer only divides this data by 9.81 m s-2.

* time series of surface pressure: assuming that the values shown are global mean values, the absolute values and the variability look wrong

I think the +9.85...e4 on the y axis means that you have to add this value to the numbers on the axis label. I think that's just a very weird way to plot this, no idea why the diagnostic does that. The raw values look totally fine:

[axel-lauer]

I successfully ran recipe_emac_new.yml and the output looks fine to me. Three quick answers to my previous comments:

• pr: OK, the preprocessor output looks fine, but I am not sure why the diagnostic expects mm/day instead of simply using the units provided in the metadata. But that's not a problem of the EMAC related additions.
• ps: also OK, the preprocessor output looks fine but I do not fully understand why the diagnostic creates such a strange looking time series. But again, this is not related to the EMAC extensions and thus not relevant here.
• zg @ 500 hPa: OK, the original EMAC output already looks weird with the 500 hPa geopotential heitght in Antarctica and Greenland being lower than the surface height! We should ask our EMAC modelers about this, but again, this is not related to the EMAC additions and thus fine.

So in summary, all fine and green light from the science review.

[remi-kazeroni]

This PR looks good to me: code, documentation, tests, recipe run with recipe_emac_new.yml. Thanks for your work @schlunma! It's nice to have support for EMAC data as part of the on-the-fly cmorizers.

[schlunma]

Awesome, great!! Many thanks for the quick review @axel-lauer @remi-kazeroni 🎉