Recipe testing and comparison for release 2.11.0 #3616

mo-gill · 2024-05-24T09:57:51Z

Test results

See https://esmvaltool.dkrz.de/shared/esmvaltool/v2.11.0rc1/.

Comparisons

Action required: 95 out of 133 recipe runs need to be inspected by a human.

See comparison_output_v2.11.0rc1.txt.

Path to the output directories on DKRZ/Levante:

/scratch/b/b382148/esmvaltool_output

System and settings

environment_v2.11.0.txt

levante6> mamba --version
mamba 1.5.8
conda 24.5.0

levante6> esmvaltool version
ESMValCore: 2.11.0rc2.dev3+g20ae705fe
ESMValTool: 2.11.0.dev78+g4ab64905c.d20240523

This is what @ehogan did on DKRZ: ESMValGroup/ESMValCore#2389 (reply in thread)

Recipe running session 2024-05-24 10:41:05.726830

Recipes that ran successfully (136 out of 160)

Click to expand

recipe_albedolandcover.yml
recipe_anav13jclim.yml
recipe_arctic_ocean.yml
recipe_autoassess_landsurface_permafrost.yml
recipe_autoassess_landsurface_soilmoisture.yml
recipe_autoassess_landsurface_surfrad.yml
recipe_autoassess_stratosphere.yml
recipe_bock20jgr_fig_1-4.yml
recipe_bock20jgr_fig_6-7.yml
recipe_capacity_factor.yml
recipe_climate_change_hotspot.yml
recipe_climwip_brunner2019_med.yml
recipe_climwip_brunner20esd.yml
recipe_climwip_test_basic.yml
recipe_climwip_test_performance_sigma.yml
recipe_clouds_bias.yml
recipe_clouds_ipcc.yml
recipe_cmug_h2o.yml
recipe_concatenate_exps.yml
recipe_consecdrydays.yml
recipe_correlation.yml
recipe_cox18nature.yml
recipe_cvdp.yml
recipe_daily_era5.yml
recipe_deangelis15nat.yml
recipe_deangelis15nat_fig1_fast.yml
recipe_decadal.yml
recipe_diurnal_temperature_index.yml
recipe_eady_growth_rate.yml
recipe_ecs.yml
recipe_ecs_constraints.yml
recipe_ecs_scatter.yml
recipe_ensclus.yml
recipe_era5-land.yml
recipe_esacci_lst.yml
recipe_esacci_oc.yml
recipe_extract_shape.yml
recipe_extreme_index.yml
recipe_eyring06jgr.yml
recipe_flato13ipcc_figure_914.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figure_942.yml
recipe_flato13ipcc_figure_945a.yml
recipe_flato13ipcc_figure_96.yml
recipe_flato13ipcc_figure_98.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figures_92_95.yml
recipe_flato13ipcc_figures_938_941_cmip3.yml
recipe_flato13ipcc_figures_938_941_cmip6.yml
recipe_galytska23jgr.yml
recipe_gier2020bg.yml
recipe_globwat.yml
recipe_heatwaves_coldwaves.yml
recipe_hydro_forcing.yml
recipe_hyint.yml
recipe_hyint_extreme_events.yml
recipe_hype.yml
recipe_iht_toa.yml
recipe_impact.yml
recipe_ipccwg1ar6ch3_fig_3_43.yml
recipe_ipccwg1ar6ch3_fig_3_9.yml
recipe_julia.yml
recipe_kcs.yml
recipe_landcover.yml
recipe_lauer13jclim.yml
recipe_lauer22jclim_fig1_clim.yml
recipe_lauer22jclim_fig1_clim_amip.yml
recipe_lauer22jclim_fig2_taylor.yml
recipe_lauer22jclim_fig2_taylor_amip.yml
recipe_lauer22jclim_fig6_interannual.yml
recipe_lauer22jclim_fig7_seas.yml
recipe_lauer22jclim_fig8_dyn.yml
recipe_lauer22jclim_fig9-11c_pdf.yml
recipe_li17natcc.yml
recipe_lisflood.yml
recipe_marrmot.yml
recipe_meehl20sciadv.yml
recipe_model_evaluation_basics.yml
recipe_model_evaluation_clouds_clim.yml
recipe_model_evaluation_clouds_cycles.yml
recipe_model_evaluation_precip_zonal.yml
recipe_modes_of_variability.yml
recipe_monitor.yml
recipe_monitor_with_refs.yml
recipe_mpqb_xch4.yml
recipe_multimodel_products.yml
recipe_my_personal_diagnostic.yml
recipe_ncl.yml
recipe_ocean_Landschuetzer2016.yml
recipe_ocean_amoc.yml
recipe_ocean_bgc.yml
recipe_ocean_example.yml
recipe_ocean_ice_extent.yml
recipe_ocean_multimap.yml
recipe_ocean_scalar_fields.yml
recipe_perfmetrics_CMIP5.yml
recipe_perfmetrics_CMIP5_4cds.yml
recipe_perfmetrics_land_CMIP5.yml
recipe_preprocessor_test.yml
recipe_psyplot.yml
recipe_pv_capacity_factor.yml
recipe_python.yml
recipe_python_for_CI.yml
recipe_quantilebias.yml
recipe_r.yml
recipe_radiation_budget.yml
recipe_rainfarm.yml
recipe_runoff_et.yml
recipe_russell18jgr.yml
recipe_schlund20esd.yml
recipe_schlund20jgr_gpp_abs_rcp85.yml
recipe_schlund20jgr_gpp_change_1pct.yml
recipe_schlund20jgr_gpp_change_rcp85.yml
recipe_sea_surface_salinity.yml
recipe_seaborn.yml
recipe_seaice.yml
recipe_seaice_drift.yml
recipe_seaice_feedback.yml
recipe_shapeselect.yml
recipe_smpi.yml
recipe_smpi_4cds.yml
recipe_snowalbedo.yml
recipe_spei.yml
recipe_tcr.yml
recipe_thermodyn_diagtool.yml
recipe_toymodel.yml
recipe_validation.yml
recipe_validation_CMIP6.yml
recipe_variable_groups.yml
recipe_weigel21gmd_figures_13_16.yml
recipe_wenzel14jgr.yml
recipe_wenzel16jclim.yml
recipe_wenzel16nat.yml
recipe_wflow.yml
recipe_williams09climdyn_CREM.yml
recipe_zmnam.yml

Recipes that failed because the diagnostic script failed (7 out of 160)

recipe_combined_indices.yml
recipe_extreme_events.yml
recipe_martin18grl.yml
recipe_miles_block.yml
recipe_miles_eof.yml
recipe_miles_regimes.yml
recipe_pcrglobwb.yml

Recipes that failed because of missing data (3 out of 160)

recipe_aod_aeronet_assess.yml
recipe_bock20jgr_fig_8-10.yml
recipe_check_obs.yml

Recipes that failed because the run took too long (4 out of 160)

recipe_eyring13jgr_12.yml
recipe_ipccwg1ar6ch3_fig_3_19.yml
recipe_ipccwg1ar6ch3_fig_3_42_a.yml
recipe_lauer22jclim_fig5_lifrac.yml

Recipes that failed because they used too much memory (2 out of 160)

recipe_ipccwg1ar6ch3_fig_3_42_b.yml
recipe_lauer22jclim_fig3-4_zonal.yml

Recipes that failed of other reasons or are still running (8 out of 160)

recipe_carvalhais14nat.yml
recipe_collins13ipcc.yml
recipe_easy_ipcc.yml
recipe_ipccwg1ar6ch3_atmosphere.yml
recipe_lauer22jclim_fig9-11ab_scatter.yml
recipe_ocean_quadmap.yml
recipe_preprocessor_derive_test.yml
recipe_tebaldi21esd.yml

Analysis of failures are detailed here ESMValGroup/ESMValCore#2421 (comment)

The text was updated successfully, but these errors were encountered:

katjaweigel · 2024-05-27T08:37:13Z

Thanks for the check!
recipe_martin18grl.yml
fails because of some issue with the provenance, looks like some check got more strict and found some before not noticed issue with filenames. I'll have a look at it.

katjaweigel · 2024-05-29T07:06:33Z

Sorry, somehow I hit the wrong button when I tried to remove a comment.

Unfortunately, the issue with recipe_martin18grl.yml is more complicated than I thought before. The reason that it cannot write the figure/provenance is, that the name of some models is not found correctly anymore in droughtindex/collect_drought_*.py. The way these python diagnostics identified the models was never really good: Since the data are previously modified by the R script (diag_save_spi.R) and forwarded through ancestors in the recipe, the information about the model name was not in the cfg automatically, so I used the information in cube [model_id or source_id in cube.metadata.attributes after iris.load]. (This was never a really good idea, because due to the different keys in the attributes, it only worked for CMIP5 and CMIP6 models). Unfortunately, now this information is not there anymore for some CMIP5 models (e.g. GISS-E2-H # ) for others it still works. The reason could be somewhere in the R script, or even already in the preprocessor? (And I'm not sure if it would not be better to improve the way the python script gets its information instead of fixing it in R).

#
cube.metadata.attributes
for GISS-E2-H
{'Conventions': 'CF-1.7', 'title': 'GISS-E2-H model output prepared for CMIP5 historical', 'software': 'Created with ESMValTool v2.11.0rc2.dev3+g20ae705fe', 'caption': 'SPI index per grid point.'}
cube.metadata.attributes
for ACCESS1.3
{'Conventions': 'CF-1.7', 'branch_time': 90945.0, 'cmor_version': '2.8.0', 'contact': 'The ACCESS wiki: http://wiki.csiro.au/confluence/display/ACCESS/Home. Contact [email protected] regarding the ACCESS coupled climate model. Contact [email protected] regarding ACCESS coupled climate model CMIP5 datasets.', 'experiment': 'historical', 'experiment_id': 'historical', 'forcing': 'GHG, Oz, SA, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFC11, CFC12, CFC113, HCFC22, HFC125, HFC134a)', 'frequency': 'mon', 'initialization_method': 1, 'institute_id': 'CSIRO-BOM', 'institution': 'CSIRO (Commonwealth Scientific and Industrial Research Organisation, Australia), and BOM (Bureau of Meteorology, Australia)', 'model_id': 'ACCESS1.3', 'modeling_realm': 'atmos', 'parent_experiment': 'pre-industrial control', 'parent_experiment_id': 'piControl', 'parent_experiment_rip': 'r1i1p1', 'physics_version': 1, 'product': 'output', 'project_id': 'CMIP5', 'realization': 1, 'references': 'See http://wiki.csiro.au/confluence/display/ACCESS/ACCESS+Publications', 'source': 'ACCESS1-3 2011. Atmosphere: AGCM v1.0 (N96 grid-point, 1.875 degrees EW x approx 1.25 degree NS, 38 levels); ocean: NOAA/GFDL MOM4p1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S, 50 levels); sea ice: CICE4.1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S); land: CABLE1.0 (1.875 degree EW x 1.25 degree NS, 6 levels', 'table_id': 'Table Amon (27 April 2011) 9c851218e3842df9a62ef38b1e2575bb', 'title': 'ACCESS1-3 model output prepared for CMIP5 historical', 'version_number': 'v20120413', 'software': 'Created with ESMValTool v2.11.0rc2.dev3+g20ae705fe', 'caption': 'SPI index per grid point.'}

katjaweigel · 2024-05-29T07:34:36Z

@lukruh I think you had a better method to forward information when using ancestors?

lukruh · 2024-05-29T07:47:42Z

I usually let the ancestors create a metadata.yml in the work folder similar to the one the preprocessor creates in the preproc folders. I can have a look on the Code later.

katjaweigel · 2024-05-29T12:45:47Z

It looks like it comes already out of the preprocessor:
The input data contain all necessary global attributes, the preprocessed data do not; instead the former global attributes are listed as attributes of the variable (pr), see file attached.
recipe_martin18grl_issue_with_attributes.txt

katjaweigel · 2024-05-29T15:14:01Z

But it only gets a problem through R: I read the preproc data, which have the attributes in the variable with R. Afterwards the variable attributes are gone, so the python diagnostic cannot pick them up anymore.
If it is only this recipe, that has issues with it, I should probably fix the way the information is forwarded.
I created the issue #3637 for it

mo-gill · 2024-06-13T14:32:28Z

Below is a summary of the comparison results (full comparison details are available in compare_v2.10.0_v2.11.0rc1_output.txt).

@ESMValGroup/esmvaltool-recipe-maintainers and @ESMValGroup/esmvaltool-coreteam we would be very grateful if you would check the recipe outputs are still correct. Tick the box and add your name next to the recipe once you've checked.

Runs with v2.11.0rc1: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.11.0rc1/
Runs with v2.10: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.10.0/
Runs with v2.9: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.9.0/
Runs with v2.8 https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0/
If the plots or data files are not shown on the recipe output webpage (this happens when provenance has not been implemented in the diagnostic script), you can still download them by clicking the 'figures' or 'data' links at the bottom of the page.

The recipes where plots are different are probably the most important to check because if the data are different but the plots still look the same the changes are probably not significant. Maybe we can refine the thresholds for when data is reported as different for a future version of the comparison tool.

Plots and data are different (39 recipes)

Only plots are different (9 recipes)

Only data are different (38 recipes)

Results are the same as v2.10.0 (38 recipes)

Click to expand

recipe_concatenate_exps.yml
recipe_deangelis15nat.yml
recipe_deangelis15nat_fig1_fast.yml
recipe_decadal.yml
recipe_ecs_scatter.yml
recipe_ensclus.yml
recipe_esacci_lst.yml
recipe_esacci_oc.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figure_945a.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figures_938_941_cmip3.yml
recipe_flato13ipcc_figures_938_941_cmip6.yml
recipe_globwat.yml
recipe_hype.yml
recipe_iht_toa.yml
recipe_ipccwg1ar6ch3_fig_3_9.yml
recipe_li17natcc.yml
recipe_lisflood.yml
recipe_mpqb_xch4.yml
recipe_my_personal_diagnostic.yml
recipe_ocean_Landschuetzer2016.yml
recipe_ocean_amoc.yml
recipe_ocean_bgc.yml
recipe_ocean_multimap.yml
recipe_ocean_scalar_fields.yml
recipe_preprocessor_test.yml
recipe_r.yml
recipe_radiation_budget.yml
recipe_sea_surface_salinity.yml
recipe_seaborn.yml
recipe_seaice.yml
recipe_seaice_feedback.yml
recipe_snowalbedo.yml
recipe_variable_groups.yml
recipe_wenzel14jgr.yml
recipe_wflow.yml
recipe_williams09climdyn_CREM.yml

Unable to compare because no reference run for v2.10.0 (8 recipes)

recipe_climate_change_hotspot.yml (didn't complete because it used too much memory, see Recipe test results for v2.10 #3463 (comment))
recipe_eyring06jgr.yml (didn't complete because it took too long, see Recipe test results for v2.10 #3463 (comment))
recipe_model_evaluation_basics.yml (new recipe)
recipe_model_evaluation_clouds_clim.yml (new recipe)
recipe_model_evaluation_clouds_cycles.yml (new recipe)
recipe_model_evaluation_precip_zonal.yml (new recipe)
recipe_psyplot.yml (was a known broken recipe for version 2.10.0), see https://esmvaltool.dkrz.de/shared/esmvaltool/v2.10.0/debug.html
recipe_seaice_drift.yml (was a known broken recipe for version 2.10.0, https://esmvaltool.dkrz.de/shared/esmvaltool/v2.10.0/debug.html)

Known broken recipes (1 recipe)

recipe_julia.yml

mo-gill · 2024-06-17T09:14:22Z

Thanks to those who have checked recipes so far! Please keep checking, we are aiming to finalise the release process on Thursday this week 🙂

schlunma · 2024-06-24T14:57:03Z

Apologies for the late answer, I was on vacation.

recipe_ecs_constraints.yml and recipe_schlund20esd.yml: the plot for the Su et al. (2014) emergent constraints is vastly different in v2.11.0. Something is very wrong here.

Pre-v2.11.0:

v2.11.0:

I have not idea what's causing this. Try to have a look at this tomorrow.

recipe_monitor_with_refs.yml: The tick labels of the time axis overlap, this can probably be fixed very easily.

schlunma · 2024-06-24T14:58:31Z

Known broken recipes (3 recipes)

recipe_psyplot.yml

recipe_psyplot.yml ran successfully, why is it listed as broken here?

ehogan · 2024-06-24T15:19:31Z

Known broken recipes (3 recipes)

recipe_psyplot.yml

recipe_psyplot.yml ran successfully, why is it listed as broken here?

Thank you for checking @schlunma! 🎉 ESMValGroup/ESMValCore#2421 (comment) suggests that this recipe ran successfully, so I believe it should instead be in the Unable to compare because no reference run for v2.10.0 section; I will update this 👍

ehogan · 2024-07-04T11:33:37Z

Closing this issue now the release has been completed 👍

mo-gill self-assigned this May 24, 2024

mo-gill added this to the v2.11.0 milestone May 24, 2024

mo-gill added the release label May 24, 2024

ehogan changed the title ~~Recipe testing and comparison for release 2.11.x~~ Recipe testing and comparison for release 2.11.0 May 24, 2024

katjaweigel closed this as completed May 29, 2024

katjaweigel reopened this May 29, 2024

katjaweigel mentioned this issue May 29, 2024

Fixing global attributes for recipe_martin18grl #3637

Closed

This was referenced Jun 25, 2024

Results of Su et al. (2014) emergent constraints are wrong #3680

Closed

Prevent overlapping time axis tick labels in monitoring recipe #3682

Merged

ehogan closed this as completed Jul 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recipe testing and comparison for release 2.11.0 #3616

Recipe testing and comparison for release 2.11.0 #3616

mo-gill commented May 24, 2024 •

edited by ehogan

Loading

katjaweigel commented May 27, 2024

katjaweigel commented May 29, 2024 •

edited

Loading

katjaweigel commented May 29, 2024

lukruh commented May 29, 2024

katjaweigel commented May 29, 2024

katjaweigel commented May 29, 2024 •

edited

Loading

mo-gill commented Jun 13, 2024 •

edited by schlunma

Loading

mo-gill commented Jun 17, 2024

schlunma commented Jun 24, 2024 •

edited

Loading

schlunma commented Jun 24, 2024 •

edited

Loading

Known broken recipes (3 recipes)

ehogan commented Jun 24, 2024

Known broken recipes (3 recipes)

ehogan commented Jul 4, 2024

Recipe testing and comparison for release 2.11.0 #3616

Recipe testing and comparison for release 2.11.0 #3616

Comments

mo-gill commented May 24, 2024 • edited by ehogan Loading

Test results

Comparisons

Path to the output directories on DKRZ/Levante:

System and settings

Recipe running session 2024-05-24 10:41:05.726830

Recipes that ran successfully (136 out of 160)

Recipes that failed because the diagnostic script failed (7 out of 160)

Recipes that failed because of missing data (3 out of 160)

Recipes that failed because the run took too long (4 out of 160)

Recipes that failed because they used too much memory (2 out of 160)

Recipes that failed of other reasons or are still running (8 out of 160)

katjaweigel commented May 27, 2024

katjaweigel commented May 29, 2024 • edited Loading

katjaweigel commented May 29, 2024

lukruh commented May 29, 2024

katjaweigel commented May 29, 2024

katjaweigel commented May 29, 2024 • edited Loading

mo-gill commented Jun 13, 2024 • edited by schlunma Loading

Plots and data are different (39 recipes)

Only plots are different (9 recipes)

Only data are different (38 recipes)

Results are the same as v2.10.0 (38 recipes)

Unable to compare because no reference run for v2.10.0 (8 recipes)

Known broken recipes (1 recipe)

mo-gill commented Jun 17, 2024

schlunma commented Jun 24, 2024 • edited Loading

schlunma commented Jun 24, 2024 • edited Loading

Known broken recipes (3 recipes)

ehogan commented Jun 24, 2024

Known broken recipes (3 recipes)

ehogan commented Jul 4, 2024

mo-gill commented May 24, 2024 •

edited by ehogan

Loading

katjaweigel commented May 29, 2024 •

edited

Loading

katjaweigel commented May 29, 2024 •

edited

Loading

mo-gill commented Jun 13, 2024 •

edited by schlunma

Loading

schlunma commented Jun 24, 2024 •

edited

Loading

schlunma commented Jun 24, 2024 •

edited

Loading