Add score tree interval #5238
Add score tree interval #5238
Conversation
This is generally difficult to test against the baseline model as it changes the resulting model eval pattern so much. The spirit of this test should be captured with the previous two tests.
kylejn27
force-pushed
the
add_score_tree_interval
branch
from
2abfab0
to
2fb7217
Compare
| @@ -1153,7 +1172,9 @@ def _dmat_init(group, **params): | |||
| evals=evals, | |||
| evals_result=evals_result, feval=feval, | |||
| verbose_eval=verbose, xgb_model=xgb_model, | |||
| callbacks=callbacks) | |||
| callbacks=callbacks, | |||
There was a problem hiding this comment.
Thanks for submitting a PR for this! I like the idea of selecting range of iterations for evaluation. But I really hope that we can implement it as a simple callback instead of parameters + callback. Might need some refactoring.
There was a problem hiding this comment.
Thanks for the feedback! I think that's a good idea, I'll see what I can do
There was a problem hiding this comment.
@trivialfis Finally have some time to get back to this.
I agree with you that it would be nice for this to be implemented as only a callback. Its entirely possible that I'm missing something obvious but I'm not sure if it can be without making significant changes to the code.
The main functionality added from this PR is the ability to skip the bst.eval_set method if the current iteration doesn't occur within the specified range (score_tree_interval).
I suppose modifying _train_internal to run bst.eval_set inside of a callback might work. This new score callback could be initialized with eval_start and eval_interval defaulting to 0 and 1 respectively. If the user wishes to change the defaults the user could manually add this callback with different initialization parameters. The only problem with this, would be that the state of evaluation_result_list would have to to be passed around between the different callbacks and the order that the callbacks are executed in would matter (this new callback would have to go first)
Happy to chat about this further, perhaps moving where the model evaluation occurs is the best solution
There was a problem hiding this comment.
@kylejn27 Let me take a deeper look.
|
New callback is landed, I will revisit this feature. |
What
Adds
eval_startto control the start iteration for model evaluation during model trainingAdds
eval_intervalto control the interval that model evaluation occurs during model trainingWhy
Implements #5090