[SPARK-12517] add default RDD name for one created via sc.textFile

[wyaron]

The feature was first added at commit: 7b877b2 but was later removed (probably by mistake) at commit: fc8b581.
This change sets the default path of RDDs created via sc.textFile(...) to the path argument.

Here is the symptom:

• Using spark-1.5.2-bin-hadoop2.6:

scala> sc.textFile("/home/root/.bashrc").name
res5: String = null

scala> sc.binaryFiles("/home/root/.bashrc").name
res6: String = /home/root/.bashrc

• while using Spark 1.3.1:

scala> sc.textFile("/home/root/.bashrc").name
res0: String = /home/root/.bashrc

scala> sc.binaryFiles("/home/root/.bashrc").name
res1: String = /home/root/.bashrc

[andrewor14]

@wyaron please file a JIRA and attach to the title of this PR. See how other patches are opened.

[andrewor14]

The changes here look OK to me.

[andrewor14]

ok to test

[SparkQA]

Test build #48271 has finished for PR 10456 at commit 86efdad.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

LGTM as well. Could you add testcases for this change?

[SparkQA]

Test build #48305 has finished for PR 10456 at commit 081a723.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48307 has finished for PR 10456 at commit fe3d87d.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48309 has finished for PR 10456 at commit a2f5f07.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48310 has finished for PR 10456 at commit 50cd119.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

Could you compare by === instead of == ?
We can get better error messages by === operator.

[SparkQA]

Test build #48312 has finished for PR 10456 at commit 825f3f3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

Indentation is broken.

[SparkQA]

Test build #48313 has finished for PR 10456 at commit 99b5bde.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wyaron]

Thanks for the quick CR !
Apparently, the new unit test has just failed for sc.wholeTextFiles(...) :-)
My guess is that it was broken in commit 297048f.
The change called setName on the WholeTextFileRDD and then called map on it which yields a new RDD without the default name set (see code snippet below).

It seems that setName should be called AFTER the map is invoked, hence on the returned RDD.
Note that wholeTextFiles provides the file name for each data item in the RDD so the defaul path name is less useful... still, for consistency purposes it should be fixed.

here is the original change:
new WholeTextFileRDD(
this,
classOf[WholeTextFileInputFormat],
- classOf[String],
- classOf[String],
+ classOf[Text],
+ classOf[Text],
updateConf,
- minPartitions).setName(path)
+ minPartitions).setName(path).map(record => (record._1.toString, record._2.toString))
}

I have committed the proposed change which moves setName after the map. I've tested locally and now the name field is valid.

does it make sense ?

[SparkQA]

Test build #48314 has finished for PR 10456 at commit 08d3fe2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

I have committed the proposed change which moves setName after the map. I've tested locally and now the name field is valid.

does it make sense ?

Yeah, I think it's good catch.

[andrewor14]

LGTM

[sarutak]

O.K, I'll merge this into master and branch-1.6. Thanks @wyaron !