diff --git a/README.md b/README.md index e7daf9d81..e13b8b07b 100644 --- a/README.md +++ b/README.md @@ -37,16 +37,16 @@ - - 2.3 - v1.1.1 - 2.4* + v2.0.0 3.0 + + 3.1 + diff --git a/benchmark/scala/pom.xml b/benchmark/scala/pom.xml index f3a36d326..18235a549 100644 --- a/benchmark/scala/pom.xml +++ b/benchmark/scala/pom.xml @@ -3,13 +3,13 @@ 4.0.0 com.microsoft.spark microsoft-spark-benchmark - 1.1.1 + 2.0.0 2019 UTF-8 2.11.8 2.11 - 2.3.2 + 2.4.4 diff --git a/docs/release-notes/2.0.0/release-2.0.0.md b/docs/release-notes/2.0.0/release-2.0.0.md new file mode 100644 index 000000000..366e80c00 --- /dev/null +++ b/docs/release-notes/2.0.0/release-2.0.0.md @@ -0,0 +1,199 @@ +# .NET for Apache Spark 2.0.0 Release Notes + +### Deprecation for Spark 2.3 + +We have officially dropped support for Spark 2.3 in the 2.0.0 release. The last Spark 2.3 release (2.3.4) was back in September 2019, and no new release is planned for Spark 2.3. Since there have been no new features introduced for Spark 2.3 in the last few releases of .NET for Apache Spark, if you are relying on Spark 2.3, you should be able to continue using .NET for Apache Spark 1.x. + +### New Features/Improvements + +* Support for Spark 3.1.0 APIs ([#886](https://github.com/dotnet/spark/pull/886), [#887](https://github.com/dotnet/spark/pull/887), [#888](https://github.com/dotnet/spark/pull/888), [#889](https://github.com/dotnet/spark/pull/889), [#890](https://github.com/dotnet/spark/pull/890), [#893](https://github.com/dotnet/spark/pull/893)) +* Support for Spark 2.4.8, 3.1.2 ([#953](https://github.com/dotnet/spark/pull/953)) +* Add `Exists(string path)` to `Microsoft.Spark.Hadoop.Fs.FileSystem` ([#909](https://github.com/dotnet/spark/pull/909)) +* Add `Version()` to `Microsoft.Spark.SparkContext` and `Microsoft.Spark.Sql.SparkSession` ([#919](https://github.com/dotnet/spark/pull/919)) +* Add Avro `ToAvro`, `FromAvro` APIs ([#805](https://github.com/dotnet/spark/pull/805)) +* Extensions.DotNet.Interactive add ENV var to control disposal of tmp dir ([#952](https://github.com/dotnet/spark/pull/952)) +* Add public APIs to access `JvmBridge`, `JvmObjectReference`, `IJvmObjectReferenceProvider` ([#951](https://github.com/dotnet/spark/pull/951)) + > This is exposed to help users interact with the JVM. It is provided with limited support and should be used with caution. + +### Bug Fixes + +* None + +### Infrastructure / Documentation / Etc. + +* None + +### Breaking Changes + +* Add 'Z' to the string format in Timestamp.ToString() to indicate UTC time ([#897](https://github.com/dotnet/spark/pull/897)) + +### Known Issues + +* Broadcast variables do not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#561](https://github.com/dotnet/spark/pull/561)) +* UDFs defined using class objects with closures does not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#619](https://github.com/dotnet/spark/pull/619)) +* In [dotnet-interactive](https://github.com/dotnet/interactive) blocking Spark methods that require external threads to unblock them does not work. ie `StreamingQuery.AwaitTermination` requires `StreamingQuery.Stop` to unblock ([#736](https://github.com/dotnet/spark/pull/736)) + +### Compatibility + +#### Backward compatibility + +The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker. + + + + + + + + + + + + +
Oldest compatible Microsoft.Spark.Worker version
v2.0.0
+ +#### Forward compatibility + +The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with. + + + + + + + + + + + + +
Oldest compatible .NET for Apache Spark release version
v2.0.0
+ +### Supported Spark Versions + +The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Spark Versionmicrosoft-spark JAR
2.4.0microsoft-spark-2-4_2.11-2.0.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.2Not supported
3.0.0microsoft-spark-3-0_2.12-2.0.0.jar
3.0.1
3.0.2
3.1.1microsoft-spark-3-1_2.12-2.0.0.jar
3.1.2
+ +### Supported Delta Versions + +The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Delta VersionMicrosoft.Spark.Extensions.Delta
0.1.02.0.0
0.2.0
0.3.0
0.4.0
0.5.0
0.6.0
0.6.1
0.7.0
0.8.0
+ +### Supported Hyperspace Versions + +The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with: + + + + + + + + + + + + + + + + + + + + + + + +
Hyperspace VersionMicrosoft.Spark.Extensions.Hyperspace
0.1.02.0.0
0.2.0
0.3.0
0.4.0
diff --git a/eng/Versions.props b/eng/Versions.props index b62227350..55b205203 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -1,7 +1,7 @@ - 1.1.1 + 2.0.0 prerelease diff --git a/src/scala/pom.xml b/src/scala/pom.xml index fbef521a3..776880634 100644 --- a/src/scala/pom.xml +++ b/src/scala/pom.xml @@ -7,7 +7,7 @@ ${microsoft-spark.version} UTF-8 - 1.1.1 + 2.0.0