[SPARK-7328] [MLlib] [PySpark] Pyspark.mllib.linalg.Vectors: Missing items #5872

MechCoder · 2015-05-03T17:56:38Z

Add

Class methods squared_dist
parse
norm
numNonzeros
copy

I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (#5775) and plan to complete support for Matrices in another PR.

AmplabJenkins · 2015-05-03T17:57:09Z

Merged build triggered.

AmplabJenkins · 2015-05-03T17:57:15Z

Merged build started.

MechCoder · 2015-05-03T17:57:37Z

ping @jkbradley @mengxr Please have a look!

SparkQA · 2015-05-03T17:58:51Z

Test build #31699 has started for PR 5872 at commit 686b0de.

AmplabJenkins · 2015-05-03T18:02:10Z

Merged build triggered.

AmplabJenkins · 2015-05-03T18:02:15Z

Merged build started.

SparkQA · 2015-05-03T18:03:49Z

Test build #31701 has started for PR 5872 at commit 350aec7.

SparkQA · 2015-05-03T19:17:51Z

Test build #31701 has finished for PR 5872 at commit 350aec7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T19:17:56Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-03T19:17:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31701/
Test FAILed.

SparkQA · 2015-05-03T19:18:53Z

Test build #31699 has finished for PR 5872 at commit 686b0de.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T19:18:58Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-03T19:18:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31699/
Test FAILed.

AmplabJenkins · 2015-05-03T19:47:09Z

Merged build triggered.

AmplabJenkins · 2015-05-03T19:47:16Z

Merged build started.

SparkQA · 2015-05-03T19:48:53Z

Test build #31707 has started for PR 5872 at commit 59909f9.

SparkQA · 2015-05-03T21:24:54Z

Test build #31707 has finished for PR 5872 at commit 59909f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-03T21:24:59Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-03T21:25:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31707/
Test PASSed.

MechCoder · 2015-05-05T19:58:16Z

@mengxr Sorry for spaming but could you have a look at this too? My exams start this Friday and hence would like to complete the linalg module at least by then.

jkbradley · 2015-05-05T20:00:43Z

I'll take a look now

jkbradley · 2015-05-05T20:23:02Z

python/pyspark/mllib/linalg.py

+        6.0
+        """
+        if p == 1:
+            return np.sum(np.abs(self.array))


You can use numpy.linalg.norm(self.array, p)

jkbradley · 2015-05-05T20:23:41Z

I just accidentally sent some comments, but I'm not done yet

AmplabJenkins · 2015-05-07T18:32:22Z

Merged build started.

SparkQA · 2015-05-07T18:32:29Z

Test build #32133 has started for PR 5872 at commit 9bd269d.

MechCoder · 2015-05-07T18:35:21Z

@jkbradley fixed!

AmplabJenkins · 2015-05-07T18:37:12Z

Merged build triggered.

AmplabJenkins · 2015-05-07T18:37:22Z

Merged build started.

SparkQA · 2015-05-07T18:39:05Z

Test build #32135 has started for PR 5872 at commit a8ff1e0.

SparkQA · 2015-05-07T19:49:39Z

Test build #32135 has finished for PR 5872 at commit a8ff1e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T19:49:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T19:49:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32135/
Test FAILed.

MechCoder · 2015-05-07T19:55:11Z

test this please

AmplabJenkins · 2015-05-07T19:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-07T19:57:18Z

Merged build started.

SparkQA · 2015-05-07T19:57:59Z

Test build #32143 has started for PR 5872 at commit a8ff1e0.

SparkQA · 2015-05-07T20:19:40Z

Test build #32133 has finished for PR 5872 at commit 9bd269d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T20:19:45Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T20:19:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32133/
Test PASSed.

jkbradley · 2015-05-07T21:01:32Z

LGTM Merging into master and branch-1.4 Thanks!

…items Add 1. Class methods squared_dist 3. parse 4. norm 5. numNonzeros 6. copy I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (#5775) and plan to complete support for Matrices in another PR. Author: MechCoder <[email protected]> Closes #5872 from MechCoder/local_linalg_api and squashes the following commits: a8ff1e0 [MechCoder] minor ce3e53e [MechCoder] Add error message for parser 1bd3c04 [MechCoder] Robust parser and removed unnecessary methods f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items (cherry picked from commit 347a329) Signed-off-by: Joseph K. Bradley <[email protected]>

MechCoder · 2015-05-07T21:06:30Z

Thanks! I have my exams from 8th to 18th. So will be away for a while, trying to study for those.. Just a heads up!

jkbradley · 2015-05-07T21:09:47Z

OK, good luck with them!

SparkQA · 2015-05-07T22:28:00Z

Test build #32143 timed out for PR 5872 at commit a8ff1e0 after a configured wait of 150m.

AmplabJenkins · 2015-05-07T22:28:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T22:28:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32143/
Test FAILed.

…items Add 1. Class methods squared_dist 3. parse 4. norm 5. numNonzeros 6. copy I made a few vectorizations wrt squared_dist and dot as well. I have added support for SparseMatrix serialization in a separate PR (apache#5775) and plan to complete support for Matrices in another PR. Author: MechCoder <[email protected]> Closes apache#5872 from MechCoder/local_linalg_api and squashes the following commits: a8ff1e0 [MechCoder] minor ce3e53e [MechCoder] Add error message for parser 1bd3c04 [MechCoder] Robust parser and removed unnecessary methods f779561 [MechCoder] [SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items

MechCoder changed the title ~~[SPARK-7328] Pyspark.mllib.linalg.Vectors: Missing items~~ [SPARK-7328] [MLlib] [PySpark] Pyspark.mllib.linalg.Vectors: Missing items May 3, 2015

MechCoder force-pushed the local_linalg_api branch from 686b0de to 350aec7 Compare May 3, 2015 17:58

MechCoder force-pushed the local_linalg_api branch from 350aec7 to 59909f9 Compare May 3, 2015 19:42

jkbradley reviewed May 5, 2015
View reviewed changes

minor

a8ff1e0

MechCoder force-pushed the local_linalg_api branch from 9bd269d to a8ff1e0 Compare May 7, 2015 18:33

asfgit closed this in 347a329 May 7, 2015

MechCoder deleted the local_linalg_api branch May 7, 2015 21:05

[SPARK-7328] [MLlib] [PySpark] Pyspark.mllib.linalg.Vectors: Missing items #5872

[SPARK-7328] [MLlib] [PySpark] Pyspark.mllib.linalg.Vectors: Missing items #5872

Conversation

MechCoder commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

MechCoder commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

SparkQA commented May 3, 2015

SparkQA commented May 3, 2015

AmplabJenkins commented May 3, 2015

AmplabJenkins commented May 3, 2015

MechCoder commented May 5, 2015

jkbradley commented May 5, 2015

jkbradley May 5, 2015

Choose a reason for hiding this comment

MechCoder May 6, 2015

Choose a reason for hiding this comment

jkbradley commented May 5, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

MechCoder commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

MechCoder commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

jkbradley commented May 7, 2015

MechCoder commented May 7, 2015

jkbradley commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015