[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation. #10234

thunterdb · 2015-12-10T00:29:27Z

Replaces a number of occurences of MLlib in the documentation that were meant to refer to the spark.mllib package instead. It should clarify for new users the difference between spark.mllib (the package) and MLlib (the umbrella project for ML in spark).

It also removes some files that I forgot to delete with #10207

jkbradley · 2015-12-10T00:38:31Z

Early comment: Maybe we should reinstate all existing .md files but replace the contents with redirection links to the relevant new parts of the guide. Websites might have links which will be broken by these changes. I guess that should include files removed in [https://github.com//pull/10207] too.

SparkQA · 2015-12-10T00:53:53Z

Test build #47461 has finished for PR 10234 at commit 42bf337.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

jkbradley · 2015-12-10T01:04:27Z

Related to my earlier comment, it might be good to rename the ml-intro file to ml-guide; there isn't a need to rename it.

jkbradley · 2015-12-10T01:08:54Z

docs/ml-classification-regression.md

 regression and linear least squares with $L_1$ or $L_2$ regularization.
 Refer to [the linear methods in mllib](mllib-linear-methods.html) for
-details.  In `spark.ml`, we also include Pipelines API for [Elastic
+details about implementation and tuning.  We also include a Dataframe API for [Elastic


"DataFrame" (F capitalized) (elsewhere too)

Corrected everywhere in the documentation

jkbradley · 2015-12-10T01:11:59Z

Done with review.
One final comment: I'd prefer adding spark.ml or spark.mllib to the end of both the title and displayTitle. I like having it in displayTitle since that provides the clearest signal about which page the user is viewing.

thunterdb · 2015-12-10T01:15:35Z

@jkbradley I agree with you. Note that when you use the doc, the current section appears in bold in the side menu (and the submenu gets expandend). This is why I did not include it again, but I do not have a strong opinion on this point:

jkbradley · 2015-12-10T01:21:16Z

OK then I'll still vote for including it : )

srowen · 2015-12-10T12:43:50Z

docs/ml-classification-regression.md

@@ -27,10 +27,10 @@ displayTitle: Classification and regression in spark.ml
 * This will become a table of contents (this text will be scraped).
 {:toc}

-In MLlib, we implement popular linear methods such as logistic
+In `spark.ml`, we implement popular linear methods such as logistic


spark.mllib right?

Not this file; the "ml-" prefix ones are for spark.ml. (It's true the functionality is almost the same currently, but it's a bit different and will diverge more.)

I see the purpose now. It was the old MLlib text, but a lot of it still applies. The distinction is removed.

thunterdb · 2015-12-10T18:41:14Z

@jkbradley done with changes, let me know what you think.

SparkQA · 2015-12-10T18:45:36Z

Test build #47530 has finished for PR 10234 at commit 6d66645.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

jkbradley · 2015-12-10T19:05:12Z

In addition to moving ml-intro back to ml-guide, it'd be nice if the sidebar had links back to the main spark.ml and spark.mllib pages. That could be done in a separate JIRA/PR, if you prefer.

thunterdb · 2015-12-10T19:20:00Z

@jkbradley done:

SparkQA · 2015-12-10T19:28:40Z

Test build #47535 has finished for PR 10234 at commit c75a5ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

SparkQA · 2015-12-10T19:51:14Z

Test build #47537 has finished for PR 10234 at commit 8432ac9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

jkbradley · 2015-12-10T20:13:32Z

docs/ml-survival-regression.md

-title: Survival Regression - ML
-displayTitle: <a href="ml-guide.html">ML</a> - Survival Regression
+title: Survival Regression - spark.ml
+displayTitle: Survival Regression - spark.ml


This doc should now be a redirect to the ml-classification-regression.html#survival-regression section.

Also, it looks like some of the math renders incorrectly, but let's fix that in a follow-up.

jkbradley · 2015-12-10T20:15:20Z

That's the only remaining issue I found. I checked against the Spark 1.5 doc links as well.

jkbradley · 2015-12-10T20:36:53Z

LGTM pending tests. Thanks! (not sure what the deal is with survival regression math; I probably am missing some library on my computer.)

SparkQA · 2015-12-10T20:40:22Z

Test build #47540 has finished for PR 10234 at commit 68f7e53.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

jkbradley · 2015-12-10T20:50:08Z

Merging with master and branch-1.6
Thank you!

…rk.mllib and mllib in the documentation. Replaces a number of occurences of `MLlib` in the documentation that were meant to refer to the `spark.mllib` package instead. It should clarify for new users the difference between `spark.mllib` (the package) and MLlib (the umbrella project for ML in spark). It also removes some files that I forgot to delete with #10207 Author: Timothy Hunter <[email protected]> Closes #10234 from thunterdb/12212. (cherry picked from commit 2ecbe02) Signed-off-by: Joseph K. Bradley <[email protected]>

thunterdb added 4 commits December 9, 2015 15:07

removed old files

edddc09

removed old files

4b328e5

last changes

62dc368

missing conversions

42bf337

jkbradley reviewed Dec 10, 2015
View reviewed changes

srowen reviewed Dec 10, 2015
View reviewed changes

thunterdb added 2 commits December 10, 2015 09:59

Frames

cb47dd1

added stubs and spark.mlxxx everywhere

6d66645

thunterdb added 2 commits December 10, 2015 11:11

put the intro again into the ml-guide

c75a5ca

links in the side bar

8432ac9

jkbradley reviewed Dec 10, 2015
View reviewed changes

changes

68f7e53

asfgit closed this in 2ecbe02 Dec 10, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation. #10234

[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation. #10234

thunterdb commented Dec 10, 2015

jkbradley commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015

jkbradley Dec 10, 2015

thunterdb Dec 10, 2015

jkbradley commented Dec 10, 2015

thunterdb commented Dec 10, 2015

jkbradley commented Dec 10, 2015

srowen Dec 10, 2015

jkbradley Dec 10, 2015

srowen Dec 10, 2015

thunterdb commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015

thunterdb commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley Dec 10, 2015

jkbradley commented Dec 10, 2015

jkbradley commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015

[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation. #10234

[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation. #10234

Conversation

thunterdb commented Dec 10, 2015

jkbradley commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015

jkbradley Dec 10, 2015

Choose a reason for hiding this comment

thunterdb Dec 10, 2015

Choose a reason for hiding this comment

jkbradley commented Dec 10, 2015

thunterdb commented Dec 10, 2015

jkbradley commented Dec 10, 2015

srowen Dec 10, 2015

Choose a reason for hiding this comment

jkbradley Dec 10, 2015

Choose a reason for hiding this comment

srowen Dec 10, 2015

Choose a reason for hiding this comment

thunterdb commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015

thunterdb commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley Dec 10, 2015

Choose a reason for hiding this comment

jkbradley commented Dec 10, 2015

jkbradley commented Dec 10, 2015

SparkQA commented Dec 10, 2015

jkbradley commented Dec 10, 2015