[SPARK-12031][Core][BUG]: Integer overflow when do sampling #10023

uncleGen · 2015-11-28T07:58:02Z

No description provided.

uncleGen · 2015-11-28T08:05:01Z

Jenkins, test this please

srowen · 2015-11-28T09:18:03Z

core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala

      val rand = new XORShiftRandom(seed)
      while (input.hasNext) {
        val item = input.next()
-        val replacementIndex = rand.nextInt(i)
+        val replacementIndex = l < Int.MaxValue match {


I think the change is fine except that the case true business strikes me as pointlessly verbose Scala. if else is much clearer.

SparkQA · 2015-11-28T09:42:05Z

Test build #46836 has finished for PR 10023 at commit 986fa5c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

uncleGen · 2015-11-28T09:52:17Z

[info] - failing to fetch classes from HTTP server should not leak resources (SPARK-6209) *** FAILED *** (1 second, 392 milliseconds)
[info] java.lang.ClassNotFoundException: ReplFakeClass2
[info] at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
[info] at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
[info] at org.apache.spark.repl.ExecutorClassLoaderSuite$$anonfun$7.apply$mcV$sp(ExecutorClassLoaderSuite.scala:148)
[info] at org.apache.spark.repl.ExecutorClassLoaderSuite$$anonfun$7.apply(ExecutorClassLoaderSuite.scala:131)
[info] at org.apache.spark.repl.ExecutorClassLoaderSuite$$anonfun$7.apply(ExecutorClassLoaderSuite.scala:131)
[info] at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
[info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
....

SparkQA · 2015-11-28T10:10:52Z

Test build #46837 has finished for PR 10023 at commit 986fa5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-11-28T12:23:56Z

core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala

+        val replacementIndex = if (l < Int.MaxValue) {
+          rand.nextInt(l.toInt)
+        } else {
+          rand.nextLong()


This isn't valid, because it chooses a value from (negative) Long.MIN_VALUE to Long.MAX_VALUE. You need to choose a number in [0, l).

Rather than even special case it, just use

val replacementIndex = (random.nextDouble() * l).toLong

SparkQA · 2015-11-28T14:11:37Z

Test build #46838 has finished for PR 10023 at commit bb2ed41.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-30T05:12:48Z

Test build #46866 has finished for PR 10023 at commit 3cafda5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-11-30T09:42:12Z

LGTM. CC @rxin for a look if possible

rxin · 2015-11-30T12:08:45Z

core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala

      val rand = new XORShiftRandom(seed)
      while (input.hasNext) {
        val item = input.next()
-        val replacementIndex = rand.nextInt(i)
+        val replacementIndex = (rand.nextDouble() * l).toLong


srowen · 2015-12-01T09:38:12Z

@mengxr I'm going to merge tomorrow to get this fix in for 1.6

mengxr · 2015-12-01T19:45:28Z

@uncleGen How many items did you have to trigger this error? We assume that there are less than Int.MaxValue items in a single partition.

uncleGen · 2015-12-02T01:26:46Z

@mengxr In my case, I do 10TB data sort in 96 partitions. When do range partition, "java.lang.IllegalArgumentException: n must be positive" exception was thrown. At that time, I saw the partition contained 10.3GB items, and the number of items exceeded the "Int.MaxValue" limit

uncleGen · 2015-12-02T01:28:51Z

@mengxr sorry, was 48 executors with 2 cores, and 1024 partitions.

uncleGen · 2015-12-04T13:47:40Z

@srowen @mengxr any problems?

srowen · 2015-12-04T14:17:11Z

@mengxr is that really an assumption that the code makes? the underlying partitions in, say, HDFS can certainly have more and @uncleGen 's reproduction seems legitimate. We could make a better error and forbid it though it seems maybe OK to support it anyway in this method?

srowen · 2015-12-07T09:33:21Z

@mengxr I'd like to merge this on the grounds that it can't hurt at the worst, and solves a problem that either is real now or could come up later.

SparkQA · 2015-12-08T14:11:01Z

Test build #2181 has finished for PR 10023 at commit 3cafda5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Author: uncleGen <[email protected]> Closes #10023 from uncleGen/1.6-bugfix. (cherry picked from commit a113216) Signed-off-by: Sean Owen <[email protected]>

srowen · 2015-12-09T15:10:16Z

Merged to master/1.6

uncleGen added 2 commits November 28, 2015 15:51

bug fix

d7eec45

code style

986fa5c

srowen reviewed Nov 28, 2015
View reviewed changes

modify some pointless and verbose code

bb2ed41

srowen reviewed Nov 28, 2015
View reviewed changes

bug fix

3cafda5

rxin reviewed Nov 30, 2015
View reviewed changes

asfgit closed this in a113216 Dec 9, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12031][Core][BUG]: Integer overflow when do sampling #10023

[SPARK-12031][Core][BUG]: Integer overflow when do sampling #10023

uncleGen commented Nov 28, 2015

uncleGen commented Nov 28, 2015

srowen Nov 28, 2015

SparkQA commented Nov 28, 2015

uncleGen commented Nov 28, 2015

SparkQA commented Nov 28, 2015

srowen Nov 28, 2015

uncleGen Nov 30, 2015

SparkQA commented Nov 28, 2015

SparkQA commented Nov 30, 2015

srowen commented Nov 30, 2015

rxin Nov 30, 2015

srowen commented Dec 1, 2015

mengxr commented Dec 1, 2015

uncleGen commented Dec 2, 2015

uncleGen commented Dec 2, 2015

uncleGen commented Dec 4, 2015

srowen commented Dec 4, 2015

srowen commented Dec 7, 2015

SparkQA commented Dec 8, 2015

srowen commented Dec 9, 2015

[SPARK-12031][Core][BUG]: Integer overflow when do sampling #10023

[SPARK-12031][Core][BUG]: Integer overflow when do sampling #10023

Conversation

uncleGen commented Nov 28, 2015

uncleGen commented Nov 28, 2015

srowen Nov 28, 2015

Choose a reason for hiding this comment

SparkQA commented Nov 28, 2015

uncleGen commented Nov 28, 2015

SparkQA commented Nov 28, 2015

srowen Nov 28, 2015

Choose a reason for hiding this comment

uncleGen Nov 30, 2015

Choose a reason for hiding this comment

SparkQA commented Nov 28, 2015

SparkQA commented Nov 30, 2015

srowen commented Nov 30, 2015

rxin Nov 30, 2015

Choose a reason for hiding this comment

srowen commented Dec 1, 2015

mengxr commented Dec 1, 2015

uncleGen commented Dec 2, 2015

uncleGen commented Dec 2, 2015

uncleGen commented Dec 4, 2015

srowen commented Dec 4, 2015

srowen commented Dec 7, 2015

SparkQA commented Dec 8, 2015

srowen commented Dec 9, 2015