Upgrade to .NET 6 #1112

AFFogarty · 2022-10-20T23:14:20Z

This PR addresses #1032 by updating to .NET 6.

cmmtchll · 2022-10-25T09:10:27Z

src/csharp/Microsoft.Spark.Worker/Command/RDDCommandExecutor.cs

@@ -112,7 +112,9 @@ internal class RDDCommandExecutor
            {
                case CommandSerDe.SerializedMode.Byte:
                    BinaryFormatter formatter = s_binaryFormatter ??= new BinaryFormatter();
+#pragma warning disable SYSLIB0011 // Type or member is obsolete


I worry about the continued use of BinaryFormatter due to https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide. pragma'ing away is necessary to compile...but I worry this needs to be fixed soon.

Yes we will need to fix this in upcoming PRs

yeah... I am worried about it, and we'll get hounded by component governance for this... but if it's a vulnerability, it's already a vulnerability now, so we may as well solve it while on .net 6

#1131 to defer

Since it's just a MemoryStream this one is really easy to fix. BinaryFormatter doesn't give any value over just writing the length and bytes.

vibhuic · 2022-10-25T12:23:00Z

We are waiting for this release as lot of our applications are in .Net 6.0 and we would like to host them in Azure Databricks, thanks for creating this PR

imback82

Any reason to include src/csharp/upgrade-assistant.clef?

AFFogarty · 2022-10-26T15:50:26Z

Any reason to include src/csharp/upgrade-assistant.clef?

Oops, removed.

AFFogarty · 2022-10-26T15:51:05Z

Need to fix the build pipeline.

dragorosson · 2022-10-28T11:53:22Z

@AFFogarty Thanks for working on this, can't wait to see it land!

…20822_net_6

vibhuic · 2022-11-02T17:54:56Z

@AFFogarty - Would be awesome if you can please merge this PR. My team is waiting for this PR ;)

AFFogarty · 2022-11-02T22:10:16Z

@AFFogarty - Would be awesome if you can please merge this PR. My team is waiting for this PR ;)

@vibhuic: We are blocked by the build failures.

chaoticsoftware · 2022-11-03T16:26:23Z

@AFFogarty any ETA on fixing the pipeline? I don't have access, but it seems we would just need to make sure the right SDK version is installed on the build machine.

Require the dotnet 6 SDK for development in this repository, to force the build process to download this SDK version.

alastairs · 2022-11-15T12:12:40Z

A bit of debugging indicates that global.json needs to be updated too--I've verified via build.cmd that results in a successful build, and have submitted AFFogarty#1 to fix the issue.

Further testing locally suggests that there may be further changes to make. Emulating the CI build produces the following output:

> .\build.cmd -pack -c Release -ci /p:PublishSparkWorker=true /p:SparkWorkerPublishDir="bin/Microsoft.Spark.Worker"
...
C:\Program Files\dotnet\sdk\7.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets(1136,5): error NETSDK1179: One of '--self-contained' or '--no-self-contained' options are required when '--runtime' is used.

This might need an update to the Arcade version, but I don't know enough about this toolchain to know how or where.

dragorosson · 2023-02-02T16:26:57Z

@AFFogarty @Niharikadutta @ericstj Is this mergeable?

ericstj

Changes look good to me, just a couple non-blocking comments.

ericstj · 2023-02-02T16:38:00Z

src/csharp/Microsoft.Spark.Worker/Command/RDDCommandExecutor.cs

@@ -112,7 +112,9 @@ internal class RDDCommandExecutor
            {
                case CommandSerDe.SerializedMode.Byte:
                    BinaryFormatter formatter = s_binaryFormatter ??= new BinaryFormatter();
+#pragma warning disable SYSLIB0011 // Type or member is obsolete


Since it's just a MemoryStream this one is really easy to fix. BinaryFormatter doesn't give any value over just writing the length and bytes.

...ons/Microsoft.Spark.Extensions.Delta.E2ETest/Microsoft.Spark.Extensions.Delta.E2ETest.csproj

eng/AfterSolutionBuild.targets

azure-pipelines.yml

GeorgeS2019 · 2023-02-09T13:56:08Z

@AFFogarty
@Niharikadutta

Please support the .NET migration to keep this Azure Doc Page VALID!

dragorosson · 2023-02-17T15:36:21Z

@Niharikadutta @suhsteve @AFFogarty Is there anything left holding this up?

GeorgeS2019 · 2023-02-17T15:38:24Z

@dragorosson
I check out the .NET6 and it works.
The issue from my side is to get the delta extension to work!

.gitignore

benchmark/README.md

examples/Microsoft.Spark.CSharp.Examples/Microsoft.Spark.CSharp.Examples.csproj

examples/Microsoft.Spark.FSharp.Examples/Microsoft.Spark.FSharp.Examples.fsproj

...ons/Microsoft.Spark.Extensions.Delta.E2ETest/Microsoft.Spark.Extensions.Delta.E2ETest.csproj

...oft.Spark.Extensions.Hyperspace.E2ETest/Microsoft.Spark.Extensions.Hyperspace.E2ETest.csproj

src/csharp/Microsoft.Spark.E2ETest/Microsoft.Spark.E2ETest.csproj

src/csharp/Microsoft.Spark.Worker.UnitTest/Microsoft.Spark.Worker.UnitTest.csproj

src/csharp/Microsoft.Spark.Worker/Microsoft.Spark.Worker.csproj

src/csharp/Microsoft.Spark.Worker/Processor/BroadcastVariableProcessor.cs

Co-authored-by: Steve Suh <[email protected]>

GeorgeS2019 · 2023-02-17T20:48:15Z

@AFFogarty
@suhsteve

Thanks for your efforts!

The following may be relevant to your Todo list to get UDF working in e.g. Notebook:

We will move away from BinaryFormatter from upcoming releases.

One user has a solution to the ToDo problem. However, I have not looked into it yet.

Additional feedback related to UDF

suhsteve

LGTM

suhsteve · 2023-02-17T21:47:13Z

src/csharp/Microsoft.Spark.Worker/Microsoft.Spark.Worker.csproj

  <ItemGroup>
    <PackageReference Include="Microsoft.DotNet.DependencyManager" Version="10.10.0-beta.20254.4" />
    <PackageReference Include="System.Memory" Version="4.5.3" />
  </ItemGroup>
-
-  <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp3.1' ">


I assume this isn't required in net 6?

Niharikadutta

LGTM, thanks @AFFogarty

chaoticsoftware · 2023-02-17T22:24:12Z

It's great to see this checked in. Thank you! I would imagine that there would be a release soon [?] @AFFogarty

GeorgeS2019 · 2023-02-17T22:26:15Z

@chaoticsoftware
It has been a long day for @AFFogarty

Let us wait till next week. Let us feedback to support them to get the .NET6 working RIGHT through nuget release.

Andrew Fogarty and others added 4 commits August 22, 2022 17:23

Migrate to .NET 6

9e308e1

Ignore serialize bug

827f9ee

Don't need the backup

4ff33bc

Merge branch 'main' into anfog/20220822_net_6

6d42fee

AFFogarty changed the title ~~[WIP] Upgrade to .NET 6~~ Upgrade to .NET 6 Oct 24, 2022

AFFogarty marked this pull request as ready for review October 24, 2022 23:02

cmmtchll reviewed Oct 25, 2022

View reviewed changes

vibhuic previously approved these changes Oct 25, 2022

View reviewed changes

Add TODO for BinaryFormatter

db31cba

AFFogarty dismissed vibhuic’s stale review via db31cba October 25, 2022 15:59

Only target .NET 6

40c2c74

cmmtchll previously approved these changes Oct 25, 2022

View reviewed changes

imback82 reviewed Oct 26, 2022

View reviewed changes

Fixed: Remove upgrade-assistant.clef

8c724ee

AFFogarty dismissed cmmtchll’s stale review via 8c724ee October 26, 2022 15:49

Andrew Fogarty added 2 commits November 1, 2022 17:25

Merge branch 'main' of https://github.com/dotnet/spark into anfog/202…

3baa15d

…20822_net_6

Fixed: Use latest windows image for .NET 6

61b7d2d

Andrew Fogarty added 3 commits November 2, 2022 15:42

AfterSolutionBuild.targets

8b40e1a

Update arcade

1f3dd3f

Update Arcade to latest version

0bc78c3

Update global.json to dotnet 6 SDK

ac1df52

Require the dotnet 6 SDK for development in this repository, to force the build process to download this SDK version.

alastairs mentioned this pull request Nov 15, 2022

Update global.json to dotnet 6 SDK AFFogarty/spark#1

Merged

AFFogarty requested a review from cutecycle January 25, 2023 15:55

cutecycle previously approved these changes Jan 26, 2023

View reviewed changes

ericstj previously approved these changes Feb 2, 2023

View reviewed changes

cutecycle mentioned this pull request Feb 2, 2023

JVM IPC Deserialization uses BinaryFormatter, which is now Deprecated for OWASP CWE #1131

Closed

Niharikadutta reviewed Feb 2, 2023

View reviewed changes

azure-pipelines.yml Show resolved Hide resolved

GeorgeS2019 mentioned this pull request Feb 9, 2023

Update .NET for Apache Spark - Azure Synapse Analytics to support Apache Spark 3.3.1 MicrosoftDocs/azure-docs#105027

Closed

Update AfterSolutionBuild.targets

385d923

AFFogarty dismissed stale reviews from ericstj and cutecycle via 385d923 February 17, 2023 19:59

AFFogarty requested review from suhsteve and Niharikadutta February 17, 2023 20:01

suhsteve reviewed Feb 17, 2023

View reviewed changes

AFFogarty and others added 4 commits February 17, 2023 15:34

Update .gitignore

d17b671

Co-authored-by: Steve Suh <[email protected]>

Add TODO

f8299c8

Add TODO

e13ac50

Add TODO

3319e0b

PR responses

5e99a8f

AFFogarty requested a review from suhsteve February 17, 2023 21:41

suhsteve approved these changes Feb 17, 2023

View reviewed changes

Niharikadutta approved these changes Feb 17, 2023

View reviewed changes

AFFogarty merged commit b63c08b into dotnet:main Feb 17, 2023

AFFogarty deleted the anfog/20220822_net_6 branch February 17, 2023 22:34

dbeavon mentioned this pull request Mar 3, 2023

.NET for Apache Spark v2.1.2 in .NET7 ? #1139

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to .NET 6 #1112

Upgrade to .NET 6 #1112

AFFogarty commented Oct 20, 2022 •

edited

Loading

cmmtchll Oct 25, 2022

Niharikadutta Nov 1, 2022

cutecycle Jan 5, 2023

cutecycle Jan 6, 2023

ericstj Feb 2, 2023

vibhuic commented Oct 25, 2022

imback82 left a comment

AFFogarty commented Oct 26, 2022

AFFogarty commented Oct 26, 2022

dragorosson commented Oct 28, 2022

vibhuic commented Nov 2, 2022

AFFogarty commented Nov 2, 2022 •

edited

Loading

chaoticsoftware commented Nov 3, 2022 •

edited

Loading

alastairs commented Nov 15, 2022 •

edited

Loading

dragorosson commented Feb 2, 2023

ericstj left a comment

ericstj Feb 2, 2023

GeorgeS2019 commented Feb 9, 2023

dragorosson commented Feb 17, 2023

GeorgeS2019 commented Feb 17, 2023 •

edited

Loading

GeorgeS2019 commented Feb 17, 2023 •

edited

Loading

suhsteve left a comment

suhsteve Feb 17, 2023

Niharikadutta left a comment

chaoticsoftware commented Feb 17, 2023

GeorgeS2019 commented Feb 17, 2023

Upgrade to .NET 6 #1112

Upgrade to .NET 6 #1112

Conversation

AFFogarty commented Oct 20, 2022 • edited Loading

cmmtchll Oct 25, 2022

Choose a reason for hiding this comment

Niharikadutta Nov 1, 2022

Choose a reason for hiding this comment

cutecycle Jan 5, 2023

Choose a reason for hiding this comment

cutecycle Jan 6, 2023

Choose a reason for hiding this comment

ericstj Feb 2, 2023

Choose a reason for hiding this comment

vibhuic commented Oct 25, 2022

imback82 left a comment

Choose a reason for hiding this comment

AFFogarty commented Oct 26, 2022

AFFogarty commented Oct 26, 2022

dragorosson commented Oct 28, 2022

vibhuic commented Nov 2, 2022

AFFogarty commented Nov 2, 2022 • edited Loading

chaoticsoftware commented Nov 3, 2022 • edited Loading

alastairs commented Nov 15, 2022 • edited Loading

dragorosson commented Feb 2, 2023

ericstj left a comment

Choose a reason for hiding this comment

ericstj Feb 2, 2023

Choose a reason for hiding this comment

GeorgeS2019 commented Feb 9, 2023

dragorosson commented Feb 17, 2023

GeorgeS2019 commented Feb 17, 2023 • edited Loading

GeorgeS2019 commented Feb 17, 2023 • edited Loading

suhsteve left a comment

Choose a reason for hiding this comment

suhsteve Feb 17, 2023

Choose a reason for hiding this comment

Niharikadutta left a comment

Choose a reason for hiding this comment

chaoticsoftware commented Feb 17, 2023

GeorgeS2019 commented Feb 17, 2023

AFFogarty commented Oct 20, 2022 •

edited

Loading

AFFogarty commented Nov 2, 2022 •

edited

Loading

chaoticsoftware commented Nov 3, 2022 •

edited

Loading

alastairs commented Nov 15, 2022 •

edited

Loading

GeorgeS2019 commented Feb 17, 2023 •

edited

Loading

GeorgeS2019 commented Feb 17, 2023 •

edited

Loading