ANTLR version conflict between Spark 2.3 and base

[jpallas]

Spark started using ANTLR 4.7 as of Spark 2.3.0, where previously it used 4.5.

BeakerX base is using version 4.5.

Some operations (any that involve parsing SQL, I imagine) trigger a problem. The ANTLR runtime complains

ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.5ANTLR Runtime version 4.7 used for parser compilation does not match the current runtime version 4.5
java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
  at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:154)
  at org.apache.spark.sql.catalyst.parser.SqlBaseLexer.<clinit>(SqlBaseLexer.java:1153)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:84)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:39)
  at org.apache.spark.sql.Column.cast(Column.scala:1017)
  ... 52 elided
Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
  ... 57 more

I don't know whether moving the base version to 4.7 would cause problems for Spark <2.3.

It looks to me like ANTLR is only being used by the Groovy and Java kernels, and is not actually used by base. So taking the dependency out of base might be a workaround.

Oh, sample problem generator:

spark.sql("")

(after loading and starting spark 2.3.1)

[scottdraves]

thanks joe.
we really need #6782