Example pyspark-inputformat for Avro file format

[ericgarcia]

This is an example showing how to map an Avro file to a pyspark RDD.

Starting Pyspark with
SPARK_CLASSPATH=examples/target/scala-2.10/spark-examples-1.1.0-SNAPSHOT-hadoop1.0.4.jar IPYTHON=1 bin/pyspark

avroRdd = sc.newAPIHadoopFile("/tmp/data.avro", 
  "org.apache.avro.mapreduce.AvroKeyInputFormat", 
  "org.apache.avro.mapred.AvroKey", 
  "org.apache.hadoop.io.NullWritable",
  keyConverter="org.apache.spark.examples.pythonconverters.AvroGenericConverter")

Note that not all data types are implemented.

Note also that null values from UNION(NULL, ...) fields will show up as the empty String "" in python.

[AmplabJenkins]

Can one of the admins verify this patch?

[MLnick]

@ericgarcia thanks for submitting this, looks good and will be helpful for others wanting to use Avro from PySpark.

Would you mind adding the python example code to the PR itself (perhaps just in the comments to the AvroGenericConverter will be fine)

[kanzhang]

How about adding case null => null for null handling?

[ericgarcia]

@kanzhang I couldn't get Java/Scala null passed into Python. It crashed when I tried to do this so I encoded it as an empty string instead. If someone else can figure this out, that would be cool.

[kanzhang]

One thing you could try to avoid NullPointerException is to wrap any intermediate value that could be null into Option(value) and only at the end of transformation call .orNull on it to get the value.

[JoshRosen]

I think that #1551 will address this issue with passing nulls from Java/Scala to Python. I'll update this thread once I've merged that PR.

[ericgarcia]

Is there a way to hand this off to anyone interested in finishing it and cleaning it up? My goal was merely to get it working for my application (accomplished!) and share the (unfinished) solution as a starting point.

[JoshRosen]

@ericgarcia If you open a new issue on the Spark JIRA and link to this pull request, I'm sure somebody will be able to pick this up and finish it. Thanks!

[ericgarcia]

Okay https://issues.apache.org/jira/browse/SPARK-2736

[JoshRosen]

Great! If you don't plan to work on this anytime soon, could you close this PR? Thanks!

[kanzhang]

@ericgarcia @JoshRosen do you mind if I give it a try?

[JoshRosen]

@kanzhang Go ahead; I've assigned the JIRA to you.