[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net #16139

sethah · 2016-12-05T05:52:33Z

What changes were proposed in this pull request?

WeightedLeastSquares now supports L1 and elastic net penalties and has an additional solver option: QuasiNewton. The docs are updated to reflect this change.

How was this patch tested?

Docs only. Generated documentation to make sure Latex looks ok.

SparkQA · 2016-12-05T06:18:06Z

Test build #69661 has finished for PR 16139 at commit c39aa6d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2016-12-05T09:17:31Z

docs/ml-advanced.md

-Unlike the original dataset which can only be stored in a distributed system,
-these statistics can be loaded into memory on a single machine if the number of features is relatively small, and then we can solve the objective function through Cholesky factorization on the driver.
+This objective function has an analytic solution and it requires only one pass over the data to collect necessary statistics to solve. For an
+$n \times m$ data matrix, these statistics require only $O(m^2)$ storage and so can be stored on a single machine when $n$ (the number of features) is


$n$ (the number of features) -> $m$ (the number of features)?

Yep, thanks!

SparkQA · 2016-12-05T12:35:21Z

Test build #69671 has finished for PR 16139 at commit 1049a6d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sethah · 2016-12-06T04:09:15Z

ping @yanboliang

yanboliang

@sethah Thanks for working on this, I left some comments.

yanboliang · 2016-12-06T10:12:28Z

docs/ml-advanced.md

@@ -59,17 +59,22 @@ Given $n$ weighted observations $(w_i, a_i, b_i)$:

 The number of features for each observation is $m$. We use the following weighted least squares formulation:
 `\[   
-minimize_{x}\frac{1}{2} \sum_{i=1}^n \frac{w_i(a_i^T x -b_i)^2}{\sum_{k=1}^n w_k} + \frac{1}{2}\frac{\lambda}{\delta}\sum_{j=1}^m(\sigma_{j} x_{j})^2
+\min_{\mathbf{x}}\frac{1}{2} \sum_{i=1}^n \frac{w_i(\mathbf{a}_i^T \mathbf{x} -b_i)^2}{\sum_{k=1}^n w_k} + \frac{1}{2}\frac{\lambda}{\delta}\sum_{j=1}^m(\sigma_{j} x_{j})^2


This formulation is out of date, we should update it to include L1/elasticNet regularization.

Great catch. Done.

yanboliang · 2016-12-06T10:14:44Z

docs/ml-advanced.md


-WeightedLeastSquares only supports L2 regularization and provides options to enable or disable regularization and standardization.
-In order to make the normal equation approach efficient, WeightedLeastSquares requires that the number of features be no more than 4096. For larger problems, use L-BFGS instead.
+Spark ML currently supports two types of solvers for the normal equations: Cholesky factorization and Quasi-Newton methods (L-BFGS/OWL-QN). Cholesky factorization


ML -> MLlib, "Spark ML" is not an official name of the component.

yanboliang · 2016-12-06T10:16:40Z

docs/ml-advanced.md

+Spark ML currently supports two types of solvers for the normal equations: Cholesky factorization and Quasi-Newton methods (L-BFGS/OWL-QN). Cholesky factorization
+depends on a positive definite covariance matrix (e.g. columns of the data matrix must be linearly independent) and will fail if this condition is violated. Quasi-Newton methods
+are still capable of providing a reasonable solution even when the covariance matrix is not positive definite, so the normal equation solver can also fall back to 
+Quasi-Newton methods in this case. This fallback is currently always enabled for the `LinearRegression` estimator.


and GeneralizedLinearRegression estimator.

yanboliang · 2016-12-06T10:28:47Z

docs/ml-advanced.md

+are still capable of providing a reasonable solution even when the covariance matrix is not positive definite, so the normal equation solver can also fall back to 
+Quasi-Newton methods in this case. This fallback is currently always enabled for the `LinearRegression` estimator.
+
+`WeightedLeastSquares` supports L1, L2, and elastic-net regularization and provides options to enable or disable regularization and standardization.


Adding following clarification should be more clear?

For L2 or no regularization, Cholesky solver is the default choice and will fall back to Quasi-Newton solver if the covariance matrix is not positive definite.

For L1/elasticNet regularization, Quasi-Newton solver is the default and only choice.

I added a note about it. It's a bit unclear to me who the audience is here. Since WLS is private, this seems more informational than anything. So I just mentioned that L1 has no analytical solution and requires QN solver. Let me know what you think.

I think the current note suffices.

The audience here is developers/contributors, and the current change is perfectly OK.

SparkQA · 2016-12-07T03:12:25Z

Test build #69766 has finished for PR 16139 at commit 2ab9675.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley

LGTM except for the one nit.

jkbradley · 2016-12-08T01:00:19Z

docs/ml-advanced.md

-WeightedLeastSquares only supports L2 regularization and provides options to enable or disable regularization and standardization.
-In order to make the normal equation approach efficient, WeightedLeastSquares requires that the number of features be no more than 4096. For larger problems, use L-BFGS instead.
+Spark MLlib currently supports two types of solvers for the normal equations: Cholesky factorization and Quasi-Newton methods (L-BFGS/OWL-QN). Cholesky factorization
+depends on a positive definite covariance matrix (e.g. columns of the data matrix must be linearly independent) and will fail if this condition is violated. Quasi-Newton methods


"e.g." -> "i.e."

Done, thanks!

SparkQA · 2016-12-08T02:09:45Z

Test build #69838 has finished for PR 16139 at commit 4931133.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yanboliang · 2016-12-08T03:40:15Z

LGTM, merged into master and branch-2.1. Thanks for all.

…or L1 and elastic-net ## What changes were proposed in this pull request? WeightedLeastSquares now supports L1 and elastic net penalties and has an additional solver option: QuasiNewton. The docs are updated to reflect this change. ## How was this patch tested? Docs only. Generated documentation to make sure Latex looks ok. Author: sethah <[email protected]> Closes #16139 from sethah/SPARK-18705. (cherry picked from commit 8225361) Signed-off-by: Yanbo Liang <[email protected]>

…or L1 and elastic-net ## What changes were proposed in this pull request? WeightedLeastSquares now supports L1 and elastic net penalties and has an additional solver option: QuasiNewton. The docs are updated to reflect this change. ## How was this patch tested? Docs only. Generated documentation to make sure Latex looks ok. Author: sethah <[email protected]> Closes apache#16139 from sethah/SPARK-18705.

update user guide

c39aa6d

MLnick reviewed Dec 5, 2016

View reviewed changes

typo

1049a6d

yanboliang reviewed Dec 6, 2016

View reviewed changes

address review

2ab9675

jkbradley reviewed Dec 8, 2016

View reviewed changes

eg to ie

4931133

asfgit closed this in 8225361 Dec 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net #16139

[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net #16139

sethah commented Dec 5, 2016

SparkQA commented Dec 5, 2016

MLnick Dec 5, 2016

sethah Dec 5, 2016

SparkQA commented Dec 5, 2016

sethah commented Dec 6, 2016

yanboliang left a comment

yanboliang Dec 6, 2016

sethah Dec 7, 2016

yanboliang Dec 6, 2016

sethah Dec 7, 2016

yanboliang Dec 6, 2016

sethah Dec 7, 2016

yanboliang Dec 6, 2016

sethah Dec 7, 2016

jkbradley Dec 8, 2016

yanboliang Dec 8, 2016

SparkQA commented Dec 7, 2016

jkbradley left a comment

jkbradley Dec 8, 2016

sethah Dec 8, 2016

SparkQA commented Dec 8, 2016

yanboliang commented Dec 8, 2016

[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net #16139

[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 and elastic-net #16139

Conversation

sethah commented Dec 5, 2016

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Dec 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 5, 2016

sethah commented Dec 6, 2016

yanboliang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 7, 2016

jkbradley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 8, 2016

yanboliang commented Dec 8, 2016