Bump Spark

[clee704]

Is there a reason that we can't bump the patch version?

I quickly checked and there was only one breaking test. Seems not serious.

[imback82]

I quickly checked and there was only one breaking test.

Interesting, which test was failing?

[clee704]

I quickly checked and there was only one breaking test.

Interesting, which test was failing?

I tried it again and this time there was no failing test.

[clee704]

Current versions:

• Spark 2.4.2
• Spark 3.0.1

Latest versions:

• Spark 2.4.7
• Spark 3.0.2

[clee704]

Can we use the latest versions? Is Azure Synapse Analytics blocking this?

[imback82]

Any reason we want to use the latest patch version? Do we need to keep track/update this every time a new patch version is released?

Do you want to check if the latest patch releases introduce any breaking changes?

[clee704]

Interesting question. I've never questioned that before. Maybe you are asking because we're not a direct user of Spark, and those bug/security fixes really don't matter for us?

In the long run, the latest patch version will most likely be the most used version among versions of the same major/minor version. So we should aim for maximum compatibility for the latest patch version. I don't think we should actively track and update for every patch version release, though. In practice, it should be okay to do this once in a while.

[MironAtHome]

3.2 is out, will test it out before tomorrow EOD.

[vishalsurana]

I'm evaluating the use of this package and I wanted to know the current status of this project. Latest Spark version is 3.5 and Scala 2.11 has been deprecated in favor of Scala 2.13 and Scala 3.

Secondly can you point me to some benchmarks that highlight the benefit of this package?