[SPARK-2736] PySpark converter and example script for reading Avro files

[kanzhang]

JIRA: https://issues.apache.org/jira/browse/SPARK-2736

This patch includes:

1. An Avro converter that converts Avro data types to Python. It handles all 3 Avro data mappings (Generic, Specific and Reflect).
2. An example Python script for reading Avro files using AvroKeyInputFormat and the converter.
3. Fixing a classloading issue.

cc @MLnick @JoshRosen @mateiz

[SparkQA]

QA tests have started for PR 1916. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18407/consoleFull

[SparkQA]

QA results for PR 1916:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class AvroWrapperToJavaConverter extends Converter[Any, Any] {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18407/consoleFull

[kanzhang]

@MLnick I put together a draft for our blog at https://github.com/kanzhang/pyspark-converter-examples. Maybe using it as a starting point?

@JoshRosen @mateiz Let us know if you have any comments or suggestions on the above draft.

[kanzhang]

SparkQA complains examples/src/main/resources/user.avsc doesn't have an Apache license. I saw other files in the same dir don't have it either. How can I exclude user.avsc from license checking?

[kanzhang]

@ericgarcia would be great if you could check if this patch works for you?

[mateiz]

Thanks for writing this up! Do you guys want to include this in 1.1? It's somewhat late in that the merge window for that has closed, but if you think it will be useful, we can probably still add it. The one part I'd like explained then is why Utils.classForName had to be used -- is it because some user classes are added at runtime and come through the Spark class loader?

[mateiz]

BTW the blog post itself looks good but perhaps more technical than you might want for a general audience. I'd suggest starting it off with some simple examples that use SequenceFiles, then adding the Avro stuff later. It's also kind of unfortunate that understanding the Avro example requires understanding Avro and its various intricacies (i.e. the generic/specific/reflect stuff, the various data types in your match statement, etc). If you wanted a simpler example you could use HBase or Cassandra there, but Avro is fine too.

[MLnick]

@kanzhang thanks for putting this together, and for the 1st draft of the blog post.

@mateiz I'd say this is not necessary for 1.1 as it's intended to be a more advanced example rather than core code - however if the Utils.classForName is a bug or issue we should perhaps fix that in a separate PR before 1.1?

I will go through the blog post and make some suggestions - from a quick read through it looks good though I agree with Matei that it could perhaps be made slightly more general.

[mateiz]

Yeah, if Utils.classForName is fixing a bug then we should definitely add it into 1.1.

[SparkQA]

QA tests have started for PR 1916. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18464/consoleFull

[SparkQA]

QA results for PR 1916:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class AvroWrapperToJavaConverter extends Converter[Any, Any] {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18464/consoleFull

[kanzhang]

@mateiz the class loading issue I encountered was when I was testing with PySpark shell. I used bin/pyspark --jars examples-jar to add the examples jar, but it still couldn't find AvroWrapperToJavaConverter. When I changed it to use Utils.classForName, it finds it. Note that this issue doesn't appear if I use bin/spark-submit --driver-class-path examples-jar ./examples/src/main/python/avro_inputformat.py command.

I personally would like to see this patch in 1.1, but I'm ok if it's not.

Thanks for your feedback on the draft. @MLnick feel free to checkout the draft and modify it any way you like.

Btw, any suggestion on what I can do to get SparkQA going?

[kanzhang]

FYI, This class loading issue has already been reported by users, see http://apache-spark-user-list.1001560.n3.nabble.com/Using-Hadoop-InputFormat-in-Python-tp12067p12092.html

We should try to squeeze this fix into 1.1 if possible.

[mateiz]

Sure, let's actually put all of this in 1.1. I didn't realize that the rest was in examples -- that's much better than putting it as an official API. Of course later we might make it an official API.

[mateiz]

BTW Jenkins had some issues recently, let me retry it

[mateiz]

Jenkins, add to whitelist and test this please

[JoshRosen]

FWIW, I don't think our test suites actually exercise our examples. We should probably manually test this, along with the other examples, during our release QA process.

[SparkQA]

QA tests have started for PR 1916. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18569/consoleFull

[SparkQA]

QA results for PR 1916:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class AvroWrapperToJavaConverter extends Converter[Any, Any] {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18569/consoleFull

[JoshRosen]

This test failed because RAT couldn't find license headers in your example avro file:

Could not find Apache license headers in the following files:
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/examples/src/main/resources/user.avsc

I'd add *.avsc to .rat-excludes so that RAT ignores avro files.

[kanzhang]

I'd add *.avsc to .rat-excludes so that RAT ignores avro files.

Ah, I see. I was trying to locate this exclude file but didn't find it. Thx.

We should probably manually test this, along with the other examples, during our release QA process.

+1. Unfortunately, we don't have a framework for adding unit tests for examples right now. When that happens, I'm happy to add couple tests.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1916. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18578/consoleFull

[SparkQA]

QA results for PR 1916:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class AvroWrapperToJavaConverter extends Converter[Any, Any] {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18578/consoleFull

[mateiz]

Alright, going to merge this. Thanks for putting this together!