[SPARK-20519][SQL][CORE]Modify to prevent some possible runtime exceptions #17796

10110346 · 2017-04-28T03:14:55Z

Signed-off-by: liuxian [email protected]

What changes were proposed in this pull request?

When the input parameter is null, may be a runtime exception occurs

How was this patch tested?

Existing unit tests

srowen

Don't make JIRAs for these; these are borderline not worth proposing, please.

srowen · 2017-04-28T07:39:46Z

core/src/main/scala/org/apache/spark/util/Utils.scala

@@ -916,7 +916,7 @@ private[spark] object Utils extends Logging {
  def setCustomHostname(hostname: String) {
    // DEBUG code
    Utils.checkHost(hostname)
-    customHostname = Some(hostname)
+    customHostname = Option(hostname)


This should be OK, but, it's already being checked for null in the line above.

10110346 · 2017-04-28T08:54:57Z

ok, thanks for review it@srowen

srowen · 2017-04-30T08:18:41Z

sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala

@@ -723,7 +723,7 @@ class SQLContext private[sql](val sparkSession: SparkSession)
   * @since 1.3.0
   */
  def tables(databaseName: String): DataFrame = {
-    Dataset.ofRows(sparkSession, ShowTablesCommand(Some(databaseName), None))
+    Dataset.ofRows(sparkSession, ShowTablesCommand(Option(databaseName), None))


CC maybe @yhuai - OK to let tables(null) act like tables()? seems reasonable

srowen · 2017-05-04T10:06:57Z

@10110346 to be conservative, maybe remove the change to SQLContext. It's not clear to me that you are supposed to be able to call {{tables(null)}} because in this case you should call {{tables()}}. A NullPointerException seems like reasonable behavior.

However I think you could also expand the improvement for Utils.checkHost and checkHostPort. They all take this "message" argument which is pretty pointless. The message is a little different in many places and doesn't include the actual value most of the time. Just setting a complete valid message once in the method rather than taking it as an argument would further help.

10110346 · 2017-05-04T10:41:06Z

I quite agree with you.
I will do it,thanks @srowen

SparkQA · 2017-05-04T14:08:56Z

Test build #3689 has finished for PR 17796 at commit 88b9e8c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-05-05T06:31:38Z

ok to test

SparkQA · 2017-05-05T06:32:37Z

Test build #76482 has started for PR 17796 at commit 0b7aac2.

10110346 · 2017-05-05T07:39:11Z

Test FAILed. I don't know why.
Environment problem? Could you help me? thanks @gatorsmile

srowen

Other than the message change, yes, looks good

srowen · 2017-05-05T11:23:51Z

core/src/main/scala/org/apache/spark/util/Utils.scala

-  def checkHost(host: String, message: String = "") {
-    assert(host.indexOf(':') == -1, message)
+  def checkHost(host: String) {
+    assert(host != null && host.indexOf(':') == -1, "Expected hostname")


The message should be s"Expected hostname (not IP) but got $host"

srowen · 2017-05-05T11:23:56Z

core/src/main/scala/org/apache/spark/util/Utils.scala

-  def checkHostPort(hostPort: String, message: String = "") {
-    assert(hostPort.indexOf(':') != -1, message)
+  def checkHostPort(hostPort: String) {
+    assert(hostPort != null && hostPort.indexOf(':') != -1, "Expected hostport")


Likewise here

SparkQA · 2017-05-06T04:24:14Z

Test build #76507 has finished for PR 17796 at commit 494928d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

10110346 · 2017-05-06T08:25:44Z

@srowen done, thank you very much.

srowen · 2017-05-06T09:37:01Z

core/src/main/scala/org/apache/spark/util/Utils.scala

-  def checkHostPort(hostPort: String, message: String = "") {
-    assert(hostPort.indexOf(':') != -1, message)
+  def checkHostPort(hostPort: String) {
+    assert(hostPort != null && hostPort.indexOf(':') != -1, s"Expected hostport but got $hostPort")


Change "hostport" to "host and port"

Signed-off-by: liuxian <[email protected]>

SparkQA · 2017-05-08T03:42:04Z

Test build #76557 has finished for PR 17796 at commit 0e879c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-05-08T09:01:26Z

Merged to master

…ptions Signed-off-by: liuxian <liu.xian3zte.com.cn> ## What changes were proposed in this pull request? When the input parameter is null, may be a runtime exception occurs ## How was this patch tested? Existing unit tests Author: liuxian <[email protected]> Closes apache#17796 from 10110346/wip_lx_0428.

srowen reviewed Apr 28, 2017

View reviewed changes

10110346 force-pushed the wip_lx_0428 branch 2 times, most recently from c5896c7 to e0eb912 Compare April 28, 2017 08:51

srowen reviewed Apr 30, 2017

View reviewed changes

10110346 force-pushed the wip_lx_0428 branch from e0eb912 to 88b9e8c Compare May 4, 2017 12:08

10110346 force-pushed the wip_lx_0428 branch from 88b9e8c to 0b7aac2 Compare May 5, 2017 01:52

srowen requested changes May 5, 2017

View reviewed changes

10110346 force-pushed the wip_lx_0428 branch from 0b7aac2 to 494928d Compare May 6, 2017 01:34

srowen reviewed May 6, 2017

View reviewed changes

Fix:Modify to prevent some possible runtime exception

0e879c8

Signed-off-by: liuxian <[email protected]>

10110346 force-pushed the wip_lx_0428 branch from 494928d to 0e879c8 Compare May 8, 2017 00:51

asfgit closed this in 0f820e2 May 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20519][SQL][CORE]Modify to prevent some possible runtime exceptions #17796

[SPARK-20519][SQL][CORE]Modify to prevent some possible runtime exceptions #17796

10110346 commented Apr 28, 2017 •

edited

Loading

srowen left a comment

srowen Apr 28, 2017

10110346 commented Apr 28, 2017

srowen Apr 30, 2017

srowen commented May 4, 2017

10110346 commented May 4, 2017

SparkQA commented May 4, 2017

gatorsmile commented May 5, 2017

SparkQA commented May 5, 2017

10110346 commented May 5, 2017 •

edited

Loading

srowen left a comment

srowen May 5, 2017

srowen May 5, 2017

SparkQA commented May 6, 2017

10110346 commented May 6, 2017

srowen May 6, 2017

SparkQA commented May 8, 2017

srowen commented May 8, 2017

[SPARK-20519][SQL][CORE]Modify to prevent some possible runtime exceptions #17796

[SPARK-20519][SQL][CORE]Modify to prevent some possible runtime exceptions #17796

Conversation

10110346 commented Apr 28, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

srowen left a comment

Choose a reason for hiding this comment

srowen Apr 28, 2017

Choose a reason for hiding this comment

10110346 commented Apr 28, 2017

srowen Apr 30, 2017

Choose a reason for hiding this comment

srowen commented May 4, 2017

10110346 commented May 4, 2017

SparkQA commented May 4, 2017

gatorsmile commented May 5, 2017

SparkQA commented May 5, 2017

10110346 commented May 5, 2017 • edited Loading

srowen left a comment

Choose a reason for hiding this comment

srowen May 5, 2017

Choose a reason for hiding this comment

srowen May 5, 2017

Choose a reason for hiding this comment

SparkQA commented May 6, 2017

10110346 commented May 6, 2017

srowen May 6, 2017

Choose a reason for hiding this comment

SparkQA commented May 8, 2017

srowen commented May 8, 2017

10110346 commented Apr 28, 2017 •

edited

Loading

10110346 commented May 5, 2017 •

edited

Loading