[SPARK-20343][BUILD] Add avro dependency in core POM to resolve build failure in SBT Hadoop 2.6 master on Jenkins

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to add

      <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
      </dependency>

in core POM to see if it resolves the build failure as below:

[error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error]     writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]     

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/2770/consoleFull

How was this patch tested?

I tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this.

Please refer the comments in #17477 (comment)

[HyukjinKwon]

cc @srowen and @vanzin. I think apparently it is a similar issue with SPARK-11538. Could you check if it makes sense?

I think this is going to resolve the problem as a safe workaround.

[HyukjinKwon]

Using Seq produces an error as below:

[error] .../spark/project/SparkBuild.scala:451: No implicit for Append.Values[Set[sbt.ModuleID], Seq[sbt.ModuleID]] found,
[error]   so Seq[sbt.ModuleID] cannot be appended to Set[sbt.ModuleID]
[error]     dependencyOverrides ++= Seq(
[error]                         ^
[error] one error found

[SparkQA]

Test build #75817 has finished for PR 17642 at commit 1ae57f2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #75820 has started for PR 17642 at commit 1ae57f2.

[HyukjinKwon]

retest this please

[srowen]

Before doing this, can you instead try adding org.apache.avro:avro as a direct dependency of core? it looks like it should be, and, could be the reason the resolution is not as expected in this case.

[SparkQA]

Test build #75821 has finished for PR 17642 at commit 1ae57f2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Sure, I will. (I was just worried of a case we should revert)

[SparkQA]

Test build #75823 has finished for PR 17642 at commit a983010.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Looks good and can't hurt, as it should be more correct. We won't know if it fixes that build until I merge though.

[HyukjinKwon]

Thank you for your approval @srowen.

[srowen]

Merged to master

[HyukjinKwon]

Ugh.. it seems this try was failed..

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/lastBuild/console

[error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error]     writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]                                                         ^

Could I give another shot with SBT one? ( and should I revert the change back in the PR I am going to open?).

[srowen]

OK, it was a decent idea. I think the change can be left in place because it's technically correct anyway.

I dislike hacking the SBT build this way, but I don't have a better idea at the moment. The hack would have to be prominently documented for sure.

[HyukjinKwon]

I see. Sure, I will. Thank you for guiding me.