Make results of recipes schlund20jgr_*.yml deterministic #2900

schlunma · 2022-10-31T11:18:31Z

Description

Major changes

Introduction of the option random_state to the MLR diagnostics which can be used to make the results of the recipe deterministic, i.e., the numerical results are identical across runs now.
Coordinate ordering in mlr/postprocess.py is now fixed to make the output of this diagnostic reproducible (see Issues related to the recipe comparison tool #2901).

--> As a result, all plots and most netCDF files can now be compared correctly with the recipe comparison tool. Note that some netCDF files are still not comparable (mainly due to differing attributes, see #2901).

Minor changes

In addition, this PR removes the usage of deprecated features and solves a bunch of Codacy issues:

Removal of deprecated sklearn.utils.metaestimators.if_delegate_has_method
Removal of deprecated sklearn.inspection.plot_partial_dependence
Removal of deprecated pandas.DataFrame.append
Removal of deprecated verbose argument for SimpleImputer
Fixed code that lead to warning about X does not have valid feature names, but SimpleImputer was fitted with feature names
Removal of GradientBoostingRegressor.loss_
Fixed Codacy issue about using raise Exception from ...
Fixed Codacy issue about using f-strings

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 Any changed dependencies have been added or removed correctly
🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

🧪 Recipe runs successfully
🧪 Recipe is well documented
🧪 Figure(s) and data look as expected from literature
🛠 Provenance information has been added

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

…ucible

valeriupredoi

thanks a lot for this @schlunma 🍺 Couple of mini bits from me - also have a look at the Codacy err too pls, it'd be nice to make that exception nicely raised like you did for the others. Gotta say, am really not understanding much of the functionality of all those changes, so I reviewed it from a purely technical/code monkey perspective. What'd be good though would be to run the thing twice and post a set of comparisons between the two runs, and if things do indeed change (plots, nc files) include the names of the ones that allow variability in the recipes' docs so next time we test (run compare via or not RTW) we'd know those are variable and they should be that way 🍺

esmvaltool/diag_scripts/mlr/custom_sklearn.py

schlunma · 2022-11-02T09:36:37Z

Thanks V for reviewing!

Regarding the Codacy issue: this one is different to the ones that I fixed. I think in this case it's fine (and necessary) to catch a general exception. It is properly re-raised or reported depending on the settings:

ESMValTool/esmvaltool/diag_scripts/mlr/custom_sklearn.py

Lines 409 to 423 in c755655

    
           except Exception as exc: 
        
               if error_score == 'raise': 
        
                   raise 
        
               if isinstance(error_score, numbers.Number): 
        
                   test_score = error_score 
        
                   warnings.warn( 
        
                       f"Estimator fit failed. The score on this train-test " 
        
                       f"partition for these parameters will be set to " 
        
                       f"{error_score:f}. Details: \n{format_exc()}", 
        
                       FitFailedWarning) 
        
               else: 
        
                   raise ValueError( 
        
                       "error_score must be the string 'raise' or a " 
        
                       "numeric value. (Hint: if using 'raise', please " 
        
                       "make sure that it has been spelled correctly.)") from exc

I ran the comparison tool on both critical recipes. In both cases no plots are affected, only netcdf mainly because of differing attributes (never because of differing data). A complete list of these issues is given here: Issues related to the recipe comparison tool #2901. Below you find a detailed list of files that still differ:

recipe_schlund20jgr_gpp_abs_rcp85.yml

Comparing recipe run(s) in:                                                                                                                                                                                                                                                    
/home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_abs_rcp85_20221031_190907                                                                                                                                                                                    
to reference in /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_abs_rcp85_20221031_163845                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
recipe_schlund20jgr_gpp_abs_rcp85.yml: results differ from reference run                                                                                                                                                                                                       
Reference run: /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_abs_rcp85_20221031_163845                                                                                                                                                                     
Current run: /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_abs_rcp85_20221031_190907                                                                                                                                                                       
Missing files:                                                                                                                                                                                                                                                                 
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_165943090559.nc                                                                                                                                                                    
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_165943107197.nc                                                                                                                                                                    
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_165943262598.nc                                                                                                                                                
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_165943289763.nc                                                                                                                                                
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_for_prediction_OBS_prediction_output_20221031_165950942347.nc                                                                                                                                         
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS_prediction_output_error_20221031_165951100339.nc                                                                                                     
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_squared_propagated_input_error_for_prediction_OBS_prediction_output_error_20221031_165951127932.nc                                                                                                    
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_for_prediction_OBS_prediction_output_20221031_165950973044.nc                                                                                                                                            
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS_prediction_output_error_20221031_165951237551.nc                                                                                                        
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_propagated_input_error_for_prediction_OBS_prediction_output_error_20221031_165951273227.nc                                                                                                       
  - work/diag_plot_mlr_models/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_165951010923.nc                                                                                                                                                                    
  - work/diag_plot_mlr_models/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_165951576160.nc                                                                                                                                                
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_165951364207.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_165951402733.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_165951445472.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_165951487461.nc                                                                                                                                           
Extra files:                                                                                                                                                                                                                                                                   
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_194139740115.nc                                                                                                                                                                    
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_194139756298.nc                                                                                                                                                                    
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_194139866639.nc                                                                                                                                                
  - work/diag_plot_cmip5_gbrt/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_194139893839.nc                                                                                                                                                
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_for_prediction_OBS_prediction_output_20221031_194159894326.nc                                                                                                                                         
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS_prediction_output_error_20221031_194200075453.nc                                                                                                     
  - work/diag_plot_mlr_models/preprocess_plot/gbr_sklearn_GPP_prediction_squared_propagated_input_error_for_prediction_OBS_prediction_output_error_20221031_194200116252.nc                                                                                                    
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_for_prediction_OBS_prediction_output_20221031_194159939901.nc                                                                                                                                            
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS_prediction_output_error_20221031_194200236226.nc                                                                                                        
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_propagated_input_error_for_prediction_OBS_prediction_output_error_20221031_194200290363.nc                                                                                                       
  - work/diag_plot_mlr_models/preprocess_plot/mmm_GPP_prediction_prediction_output_20221031_194159974934.nc                                                                                                                                                                    
  - work/diag_plot_mlr_models/preprocess_plot/mmm_GPP_squared_prediction_error_prediction_output_error_20221031_194200583571.nc                                                                                                                                                
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_194200363593.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_194200414215.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_194200454019.nc                                                                                                                                           
  - work/diag_plot_mlr_models/preprocess_plot/standard_error_of_emergent_constraint_prediction_output_error_20221031_194200495287.nc
Differing files:                                                                                                                                                                                                                                                               
  - work/diag_co2_amplitude_sensitivity/plot_co2_amplitudes/xy_feature___CESM1-BGC.nc                                                                                                                                                                                          
  - work/diag_co2_amplitude_sensitivity/plot_co2_amplitudes/xy_feature___GFDL-ESM2M.nc                                                                                                                                                                                         
  - work/diag_co2_amplitude_sensitivity/preprocess/CMIP5_CESM1-BGC_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_trend_relative_to_ref_1979-2019_feature.nc                                                                                                                          
  - work/diag_co2_amplitude_sensitivity/preprocess/CMIP5_GFDL-ESM2M_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_trend_relative_to_ref_1979-2019_feature.nc                                                                                                                         
  - work/diag_co2_amplitude_sensitivity/preprocess_for_plot/CMIP5_CESM1-BGC_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_1979-2019_feature.nc                                                                                                                                       
  - work/diag_co2_amplitude_sensitivity/preprocess_for_plot/CMIP5_GFDL-ESM2M_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_1979-2019_feature.nc                                                                                                                                      
  - work/diag_co2_cycle/plot_cycle/xy_feature___CESM1-BGC.nc                                                                                                                                                                                                                   
  - work/diag_co2_cycle/plot_cycle/xy_feature___GFDL-ESM2M.nc                                                                                                                                                                                                                  
  - work/diag_gbrt_1d/mlr/gbr_sklearn_GPP_prediction_for_prediction_OBS.nc                                                                                                                                                                                                     
  - work/diag_gbrt_1d/mlr/gbr_sklearn_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS.nc                                                                                                                                                                       
  - work/diag_gbrt_1d/mlr/gbr_sklearn_GPP_prediction_squared_propagated_input_error_for_prediction_OBS.nc                                                                                                                                                                      
  - work/diag_gbrt_1d/postprocess/gbr_sklearn_GPP_prediction_for_prediction_OBS.nc                                                                                                                                                                                             
  - work/diag_gbrt_1d/postprocess/gbr_sklearn_GPP_prediction_for_prediction_OBS_error_estimated.nc                                                                                                                                                                             
  - work/diag_gbrt_1d/postprocess/gbr_sklearn_GPP_prediction_for_prediction_OBS_error_lower_bound.nc                                                                                                                                                                           
  - work/diag_gbrt_1d/postprocess/gbr_sklearn_GPP_prediction_for_prediction_OBS_error_upper_bound.nc                                                                                                                                                                           
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_prediction_for_prediction_CESM1-BGC_of_group_CESM1-BGC.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_CESM1-BGC.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_prediction_residual_for_prediction_CESM1-BGC_of_group_CESM1-BGC.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_prediction_for_prediction_CanESM2_of_group_CanESM2.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_CanESM2.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_prediction_residual_for_prediction_CanESM2_of_group_CanESM2.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_prediction_for_prediction_GFDL-ESM2M_of_group_GFDL-ESM2M.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_GFDL-ESM2M.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_prediction_residual_for_prediction_GFDL-ESM2M_of_group_GFDL-ESM2M.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_prediction_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_HadGEM2-ES.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_prediction_residual_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_prediction_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_MIROC-ESM.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_prediction_residual_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_prediction_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_MPI-ESM-LR.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_prediction_residual_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_prediction_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_prediction_for_prediction_OBS_of_group_NorESM1-ME.nc
  - work/diag_gbrt_1d_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_prediction_residual_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_prediction_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_prediction_squared_propagated_input_error_for_prediction_OBS.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_prediction_for_prediction_OBS.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_prediction_for_prediction_OBS_error_estimated.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_prediction_for_prediction_OBS_error_lower_bound.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_prediction_for_prediction_OBS_error_upper_bound.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___GBRT-1D-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___GBRT-1D-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___GBRT-1D-prediction_output___LASSO.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___GBRT-1D-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___GBRT-1D-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_prediction_output___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot/map_prediction_output___LASSO.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___GBRT-1D-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___GBRT-1D-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___GBRT-1D-prediction_output___LASSO.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___GBRT-1D-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___GBRT-1D-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___GBRT-1D-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___GBRT-1D-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___GBRT-1D-prediction_output___LASSO.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___GBRT-1D-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___GBRT-1D-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___GBRT-1D-prediction_output_error___GBRT.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___GBRT-1D-prediction_output_error___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___GBRT-1D-prediction_output_error___LASSO.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___GBRT-1D-prediction_output_error___MMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___GBRT-1D-prediction_output_error___rMMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___LASSO-prediction_output_error___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___LASSO-prediction_output_error___GBRT.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___LASSO-prediction_output_error___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___LASSO-prediction_output_error___MMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_abs_bias_prediction_output_error___LASSO-prediction_output_error___rMMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_prediction_output_error___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_prediction_output_error___LASSO.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___GBRT-1D-prediction_output_error___GBRT.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___GBRT-1D-prediction_output_error___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___GBRT-1D-prediction_output_error___LASSO.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___GBRT-1D-prediction_output_error___MMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___GBRT-1D-prediction_output_error___rMMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___LASSO-prediction_output_error___GBRT-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___LASSO-prediction_output_error___GBRT.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___LASSO-prediction_output_error___LASSO-1D.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___LASSO-prediction_output_error___MMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_rel_bias_prediction_output_error___LASSO-prediction_output_error___rMMM.nc
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_for_prediction_OBS_prediction_output.nc
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_mlr_model_error_estim_for_prediction_OBS_prediction_output_error.nc                                                                                                                             
  - work/diag_plot_mlr_models/preprocess_plot/lasso_cv_GPP_prediction_squared_propagated_input_error_for_prediction_OBS_prediction_output_error.nc

recipe_schlund20jgr_gpp_change_rcp85.yml

Comparing recipe run(s) in:                                                                                                                                                                                                                                                    
/home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_change_rcp85_20221031_165555                                                                                                                                                                                 
to reference in /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_change_rcp85_20221031_164134                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
recipe_schlund20jgr_gpp_change_rcp85.yml: results differ from reference run                                                                                                                                                                                                    
Reference run: /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_change_rcp85_20221031_164134                                                                                                                                                                  
Current run: /home/b/b309141/scratch/esmvaltool_output/recipe_schlund20jgr_gpp_change_rcp85_20221031_165555                                                                                                                                                                    
Differing files:                                                                                                                                                                                                                                                               
  - work/diag_co2_amplitude_sensitivity/plot_co2_amplitudes/xy_feature___CanESM2.nc                                                                                                                                                                                            
  - work/diag_co2_amplitude_sensitivity/plot_co2_amplitudes/xy_feature___MIROC-ESM.nc                                                                                                                                                                                          
  - work/diag_co2_amplitude_sensitivity/plot_co2_amplitudes/xy_feature___MPI-ESM-LR.nc                                                                                                                                                                                         
  - work/diag_co2_amplitude_sensitivity/preprocess/CMIP5_CanESM2_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_trend_relative_to_ref_1979-2019_feature.nc                                                                                                                            
  - work/diag_co2_amplitude_sensitivity/preprocess/CMIP5_MIROC-ESM_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_trend_relative_to_ref_1979-2019_feature.nc                                                                                                                          
  - work/diag_co2_amplitude_sensitivity/preprocess/CMIP5_MPI-ESM-LR_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_trend_relative_to_ref_1979-2019_feature.nc                                                                                                                         
  - work/diag_co2_amplitude_sensitivity/preprocess_for_plot/CMIP5_CanESM2_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_1979-2019_feature.nc                                                                                                                                         
  - work/diag_co2_amplitude_sensitivity/preprocess_for_plot/CMIP5_MIROC-ESM_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_1979-2019_feature.nc                                                                                                                                       
  - work/diag_co2_amplitude_sensitivity/preprocess_for_plot/CMIP5_MPI-ESM-LR_Amon_esmHistorical-esmrcp85_r1i1p1_co2s_1979-2019_feature.nc                                                                                                                                      
  - work/diag_co2_cycle/plot_cycle/xy_feature___CanESM2.nc
  - work/diag_co2_cycle/plot_cycle/xy_feature___MIROC-ESM.nc
  - work/diag_co2_cycle/plot_cycle/xy_feature___MPI-ESM-LR.nc
  - work/diag_gbrt/mlr/gbr_sklearn_GPP_change_prediction_squared_propagated_input_error_for_prediction_OBS.nc
  - work/diag_gbrt/postprocess/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_error_estimated.nc
  - work/diag_gbrt/postprocess/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_error_lower_bound.nc
  - work/diag_gbrt/postprocess/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_error_upper_bound.nc
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_change_prediction_for_prediction_CESM1-BGC_of_group_CESM1-BGC.nc                                                                                                                               
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_CESM1-BGC.nc                                                                                                                                     
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CESM1-BGC/gbr_sklearn_GPP_change_prediction_residual_for_prediction_CESM1-BGC_of_group_CESM1-BGC.nc                                                                                                                      
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_change_prediction_for_prediction_CanESM2_of_group_CanESM2.nc                                                                                                                                     
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_CanESM2.nc
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/CanESM2/gbr_sklearn_GPP_change_prediction_residual_for_prediction_CanESM2_of_group_CanESM2.nc                                                                                                                            
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_change_prediction_for_prediction_GFDL-ESM2M_of_group_GFDL-ESM2M.nc                                                                                                                            
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_GFDL-ESM2M.nc                                                                                                                                   
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/GFDL-ESM2M/gbr_sklearn_GPP_change_prediction_residual_for_prediction_GFDL-ESM2M_of_group_GFDL-ESM2M.nc                                                                                                                   
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_change_prediction_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc                                                                                                                            
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_HadGEM2-ES.nc                                                                                                                                   
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/gbr_sklearn_GPP_change_prediction_residual_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc                                                                                                                   
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_change_prediction_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc                                                                                                                               
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_MIROC-ESM.nc                                                                                                                                     
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/gbr_sklearn_GPP_change_prediction_residual_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc                                                                                                                      
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_change_prediction_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc                                                                                                                            
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_MPI-ESM-LR.nc                                                                                                                                   
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/gbr_sklearn_GPP_change_prediction_residual_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_change_prediction_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_change_prediction_for_prediction_OBS_of_group_NorESM1-ME.nc
  - work/diag_gbrt_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/gbr_sklearn_GPP_change_prediction_residual_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_lime_importance___GPP_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_lime_importance___LAI_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_lime_importance___PR_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_lime_importance___RSDS_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_lime_importance___T_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_squared_mlr_model_error_estim_for_prediction_OBS.nc
  - work/diag_lasso/mlr/lasso_cv_GPP_change_prediction_squared_propagated_input_error_for_prediction_OBS.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_change_prediction_for_prediction_OBS.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_change_prediction_for_prediction_OBS_error_estimated.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_change_prediction_for_prediction_OBS_error_lower_bound.nc
  - work/diag_lasso/postprocess/lasso_cv_GPP_change_prediction_for_prediction_OBS_error_upper_bound.nc
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/lasso_cv_GPP_change_prediction_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc                                                                                                                              
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/lasso_cv_GPP_change_prediction_for_prediction_OBS_of_group_HadGEM2-ES.nc                                                                                                                                     
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/HadGEM2-ES/lasso_cv_GPP_change_prediction_residual_for_prediction_HadGEM2-ES_of_group_HadGEM2-ES.nc                                                                                                                     
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/lasso_cv_GPP_change_prediction_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc                                                                                                                                 
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/lasso_cv_GPP_change_prediction_for_prediction_OBS_of_group_MIROC-ESM.nc                                                                                                                                       
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MIROC-ESM/lasso_cv_GPP_change_prediction_residual_for_prediction_MIROC-ESM_of_group_MIROC-ESM.nc                                                                                                                        
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/lasso_cv_GPP_change_prediction_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc                                                                                                                              
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/lasso_cv_GPP_change_prediction_for_prediction_OBS_of_group_MPI-ESM-LR.nc                                                                                                                                     
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/MPI-ESM-LR/lasso_cv_GPP_change_prediction_residual_for_prediction_MPI-ESM-LR_of_group_MPI-ESM-LR.nc                                                                                                                     
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/lasso_cv_GPP_change_prediction_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc                                                                                                                              
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/lasso_cv_GPP_change_prediction_for_prediction_OBS_of_group_NorESM1-ME.nc                                                                                                                                     
  - work/diag_lasso_pseudo_reality/mlr_pseudo_reality/NorESM1-ME/lasso_cv_GPP_change_prediction_residual_for_prediction_NorESM1-ME_of_group_NorESM1-ME.nc                                                                                                                     
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_abs_bias_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_prediction_output___LASSO.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_ratio_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___GBRT.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___MMM.nc
  - work/diag_plot_mlr_models/plot/map_rel_bias_prediction_output___LASSO-prediction_output___rMMM.nc
  - work/diag_plot_mlr_models/plot_errors/map_prediction_output_error___GBRT.nc
  - work/diag_plot_mlr_models/plot_errors/map_prediction_output_error___LASSO.nc

Co-authored-by: Valeriu Predoi <[email protected]>

valeriupredoi · 2022-11-02T13:12:15Z

perfect, many thanks, Manu! All fine by me then, cheers 🍺 Do you need a sci reviewer too? I don't think so since you're the jack of all trades for these recipes, if I'm enough for the review then pls let me know and I'll merge this beast 😁

schlunma · 2022-11-02T13:14:44Z

Cheers 🍻 no need for a scientific review, results look exactly like expected!

valeriupredoi · 2022-11-02T13:15:07Z

BTW - forgot about it - you reckon any documentation is warranted for the new behaviour?

schlunma · 2022-11-02T13:17:09Z

Documentation has been added in the form of a docstring that is shown on RTD. Just search for "random_state" here (https://esmvaltool--2900.org.readthedocs.build/en/2900/api/esmvaltool.diag_scripts.mlr/models.html#optional-parameters-for-class-initialization) 😁

valeriupredoi · 2022-11-02T13:20:01Z

perfect, cheers, bud! 🍺 I'll merge when tests have finished 👍

schlunma added 4 commits October 31, 2022 12:10

Allowed usage of random_state in MLR diags to make output reproducible

fda1c7f

Removed usage of deprecated features of sklearn

084c3f3

Optimized usage of getfullargspec

ffac38d

Fixed Codacy issues

1493a4e

schlunma added the enhancement label Oct 31, 2022

schlunma added this to the v2.8.0 milestone Oct 31, 2022

schlunma self-assigned this Oct 31, 2022

schlunma added 6 commits October 31, 2022 14:44

Added random_state to schlund20jgr recieps

71dcabd

Removed warning about missing features for SimpleImputer

42be960

Removed deprecated use of 'verbose' for SimpleImputer

547c070

Removed deprecated use of pandas.DataFrame.append

dc6ce8d

Removed deprecated use of GradientBoostingRegressor.loss_

1807a44

Better warning message

839caa9

schlunma changed the title ~~Make results of recipes schlund20jgr_*.yml reproducible~~ Make results of recipes schlund20jgr_*.yml deterministic Oct 31, 2022

schlunma added 4 commits October 31, 2022 16:25

Fixed small bug in plot_1d function

f7a86a1

Made results of LIME deterministic

9adb826

Fixed order of coordinates in postprocessing.py to make output reprod…

a5e8dbb

…ucible

Fixed last warning about missing feature names

c755655

schlunma marked this pull request as ready for review October 31, 2022 20:22

schlunma requested a review from valeriupredoi October 31, 2022 20:22

valeriupredoi approved these changes Nov 1, 2022

View reviewed changes

esmvaltool/diag_scripts/mlr/custom_sklearn.py Show resolved Hide resolved

esmvaltool/diag_scripts/mlr/custom_sklearn.py Outdated Show resolved Hide resolved

schlunma and others added 2 commits November 2, 2022 10:38

Update esmvaltool/diag_scripts/mlr/custom_sklearn.py

67a9ebc

Co-authored-by: Valeriu Predoi <[email protected]>

Merge remote-tracking branch 'public/main' into deterministic_mlr

3274d8c

valeriupredoi merged commit 97878de into main Nov 2, 2022

valeriupredoi deleted the deterministic_mlr branch November 2, 2022 13:38

schlunma mentioned this pull request Jan 4, 2023

pin shapely <2.0.0 (temporary fix) so the package builds and installs correctly conda-forge/esmvaltool-suite-feedstock#14

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make results of recipes schlund20jgr_*.yml deterministic #2900

Make results of recipes schlund20jgr_*.yml deterministic #2900

schlunma commented Oct 31, 2022 •

edited

Loading

valeriupredoi left a comment •

edited

Loading

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

Make results of recipes schlund20jgr_*.yml deterministic #2900

Make results of recipes schlund20jgr_*.yml deterministic #2900

Conversation

schlunma commented Oct 31, 2022 • edited Loading

Description

Major changes

Minor changes

Before you get started

Checklist

New or updated recipe/diagnostic

valeriupredoi left a comment • edited Loading

Choose a reason for hiding this comment

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

schlunma commented Nov 2, 2022

valeriupredoi commented Nov 2, 2022

schlunma commented Oct 31, 2022 •

edited

Loading

valeriupredoi left a comment •

edited

Loading