[feature proposal] Load best model after early stopping

[PhilipMay]

Could you please implement a way to load the best model after ealy stopping? LightGBM is also doing this by default. See here: https://lightgbm.readthedocs.io/en/latest/Python-Intro.html#early-stopping

That would be a great improvement to my hyperparameter optimization workflow.

Thanks
Philip

[trivialfis]

Seems to be a nice feature. Will look into this in the future. @hcho3 WDYT?

[hcho3]

@trivialfis @PhilipMay Can you clarify what you mean? Currently, the scikit-learn interface automatically uses the best iteration when predicting:

xgboost/python-package/xgboost/sklearn.py

Lines 419 to 421 in 1fc37e4

	ntree_limit : int
	Limit number of trees in the prediction; defaults to best_ntree_limit if defined
	(i.e. it has been trained with early stopping), otherwise 0 (use all trees).

[PhilipMay]

@hcho3 This is very interesting. So with the scikit-learn interface it should work the way I would like it to work (load or use the best model). But what about the situation where you use the Booster class and do not use the scikit-learn interface? In the Booster class it just says:

Limit number of trees in the prediction; defaults to 0 (use all trees).

See here:

xgboost/python-package/xgboost/core.py

Line 1190 in 1fc37e4

Limit number of trees in the prediction; defaults to 0 (use all trees).

For me this means that it does not use the best model from early stopping.

[hcho3]

Yes, it appears that currently you need to use the scikit-learn interface to do what you want. (And use pickle to save XGBClassifier / XGBRegressor objects.) It would be nice to add num_iteration option to save_model() function so that you can truncate the model at the time of serializing.

[PhilipMay]

Yes, it appears that currently you need to use the scikit-learn interface to do what you want. (And use pickle to save XGBClassifier / XGBRegressor objects.) It would be nice to add num_iteration option to save_model() function so that you can truncate the model at the time of serializing.

Yes - and it would be nice to bring scikit-learn interface and the Booster interface to a consistent state where they both bring the best results form early stopping by default.

Or this this just a documentation bug?

[hcho3]

The issue is that only the scikit-learn interface saves the best number of round; the Booster object does not save it. As I said, it would be cleaner to simply truncate the model at the time of serializing (save_model()) to achieve consistency.