Improve QN solver stopping conditions (logistic regression) to match sklearn closer #3766
Conversation
|
Can one of the admins verify this patch? |
1 similar comment
|
Can one of the admins verify this patch? |
|
Perhaps, I'd like @tfeher have a look at this before. |
|
FYI, the are a few more differences between cuML and sklearn, which haven't been addressed here directly:
|
achirkin
changed the title
|
With our rather big default
The blue line is ours (visually not as bad as sklearn version, yet there is clearly a room for improvement :). The question is, whether there is an improvement in accuracy from doing iterations 200-600 and whether it's worth the time. |
achirkin
changed the title
|
This one with
This also changed the minimum correlation(cuML weights, sklearn weights) from |
tfeher
added
improvement
achirkin
force-pushed
the
fix-qn-stopping-condition
branch
from
d6702eb
to
69eae38
Compare
|
Removed the API-changing commits to add them in a separate PR. |
achirkin
changed the title
|
add to allowlist |
achirkin
added
3 - Ready for Review
Codecov Report
@@ Coverage Diff @@
## branch-0.20 #3766 +/- ##
==============================================
Coverage ? 86.04%
==============================================
Files ? 225
Lines ? 17117
Branches ? 0
==============================================
Hits ? 14729
Misses ? 2388
Partials ? 0
Flags with carried forward coverage won't be shown. Click here to find out more. Continue to review full report at Codecov.
|
tfeher
added
5 - Ready to Merge
|
@gpucibot merge |
…sklearn closer (rapidsai#3766) Change the QN solver (logistic regression) stopping conditions to avoid early stops in some cases (rapidsai#3645): - primary: ``` || f' ||_inf <= fmag * param.epsilon ``` - secondary: ``` |f - f_prev| <= fmag * param.delta ``` where `fmag = max(|f|, param.epsilon)`. Also change the default value of `tol` in QN solver (which sets `param.delta`) to be consistent (`1e-4`) with the logistic regression solver. #### Background The original primary stopping condition is inconsistent with the sklearn reference implementation and is often triggered too early: ``` || f' ||_2 <= param.epsilon * max(1.0, || x ||_2) ``` Here are the sklearn conditions for reference: - primary: ``` || grad f ||_inf <= gtol ``` - secondary: ``` |f - f_prev| <= ftol * max(|f|, |f_prev|, 1.0) ``` where `gtol` is and exposed parameter like `param.epsilon`, and `ftol = 2.2e-9` (hardcoded). In addition, `f` in sklearn is scaled with the sample size (softmax or sigmoid over the dataset), so it's not exactly comparable to cuML version. Currently, cuML checks the gradient w.r.t. the logistic regression weights `x`. As a result, the tolerance value goes up with the number of classes and features; the model stops too early and stays underfit. This may in part be a reason for rapidsai#3645. In this proposal I change the stopping condition to be closer to the sklearn version, but compromise the consistency with sklearn for better scaling (tolerance scales with the absolute values of the objective function). Without this scaling sklearn version seems to often run till the maximum iteration limit is reached. Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Tamas Bela Feher (https://github.com/tfeher) URL: rapidsai#3766
Change the QN solver (logistic regression) stopping conditions to avoid early stops in some cases (#3645):
where
fmag = max(|f|, param.epsilon).Also change the default value of
tolin QN solver (which setsparam.delta) to be consistent (1e-4) with the logistic regression solver.Background
The original primary stopping condition is inconsistent with the sklearn reference implementation and is often triggered too early:
Here are the sklearn conditions for reference:
where
gtolis and exposed parameter likeparam.epsilon, andftol = 2.2e-9(hardcoded).In addition,
fin sklearn is scaled with the sample size (softmax or sigmoid over the dataset), so it's not exactly comparable to cuML version.Currently, cuML checks the gradient w.r.t. the logistic regression weights
x. As a result, the tolerance value goes up with the number of classes and features; the model stops too early and stays underfit. This may in part be a reason for #3645.In this proposal I change the stopping condition to be closer to the sklearn version, but compromise the consistency with sklearn for better scaling (tolerance scales with the absolute values of the objective function). Without this scaling sklearn version seems to often run till the maximum iteration limit is reached.