Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to .NET 6 #1112

Merged
merged 48 commits into from
Feb 17, 2023
Merged

Upgrade to .NET 6 #1112

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
9e308e1
Migrate to .NET 6
Aug 22, 2022
827f9ee
Ignore serialize bug
Oct 20, 2022
4ff33bc
Don't need the backup
Oct 20, 2022
6d42fee
Merge branch 'main' into anfog/20220822_net_6
AFFogarty Oct 24, 2022
db31cba
Add TODO for BinaryFormatter
Oct 25, 2022
40c2c74
Only target .NET 6
Oct 25, 2022
8c724ee
Fixed: Remove upgrade-assistant.clef
Oct 26, 2022
3baa15d
Merge branch 'main' of https://github.com/dotnet/spark into anfog/202…
Nov 1, 2022
61b7d2d
Fixed: Use latest windows image for .NET 6
Nov 1, 2022
8b40e1a
AfterSolutionBuild.targets
Nov 2, 2022
1f3dd3f
Update arcade
Nov 2, 2022
0bc78c3
Update Arcade to latest version
Nov 2, 2022
ac1df52
Update global.json to dotnet 6 SDK
alastairs Nov 15, 2022
eacc5bd
Find-and-replace netcoreapp3.1 -> net6.0
alastairs Nov 15, 2022
7f09045
Merge pull request #1 from alastairs/patch-1
AFFogarty Nov 15, 2022
25faee9
Revert "Update Arcade to latest version"
Nov 15, 2022
ba3da68
Revert "Update arcade"
Nov 15, 2022
738a3bd
Don't compile examples with 4.6.1
Nov 15, 2022
5a6bdb3
Worker is self-contained
Nov 15, 2022
a4c7697
Selfcontained false
Nov 15, 2022
ad49093
Fixed: UseDotNet 6.x
Nov 15, 2022
a94e011
Use netcoreapp3.1 for backward-compatible testing
alastairs Nov 17, 2022
8dbf50b
Merge pull request #2 from alastairs/patch-2
AFFogarty Nov 29, 2022
723bafb
Worker is self-contained
Nov 29, 2022
94df255
Address comments on worker
Dec 6, 2022
c70d6a7
Fix typo
Dec 6, 2022
c7029e0
Switch back to self-contained for now
Dec 7, 2022
fc30c95
Revert "Switch back to self-contained for now"
ericstj Dec 8, 2022
7720a85
Specify --self-contained when publishing worker
ericstj Dec 8, 2022
00ba010
Merge pull request #3 from ericstj/anfog/20220822_net_6
AFFogarty Dec 8, 2022
ceedec1
Disable backwards compatibility for now
Dec 8, 2022
e0c7a6d
Revert backwardscompatibleframework
Dec 8, 2022
7dc36b3
Merge branch 'anfog/20220822_net_6' of https://github.com/AFFogarty/s…
Dec 8, 2022
c92cdbb
Update docs to reference .NET 6
Dec 8, 2022
f145223
Disable forwards compatibility tests for ow
Dec 8, 2022
ca64c80
Fix markdown link
Dec 9, 2022
d845910
Fix CLI name
Dec 9, 2022
9c8fbde
Merge branch 'main' into anfog/20220822_net_6
AFFogarty Jan 6, 2023
e7de72a
Merge branch 'main' into anfog/20220822_net_6
AFFogarty Jan 23, 2023
83ad587
Merge branch 'main' into anfog/20220822_net_6
AFFogarty Jan 25, 2023
e72afd5
Fixed: Disable backwards compat on new versions
Jan 25, 2023
324c454
Docs updates
Jan 25, 2023
385d923
Update AfterSolutionBuild.targets
AFFogarty Feb 17, 2023
d17b671
Update .gitignore
AFFogarty Feb 17, 2023
f8299c8
Add TODO
AFFogarty Feb 17, 2023
e13ac50
Add TODO
AFFogarty Feb 17, 2023
3319e0b
Add TODO
AFFogarty Feb 17, 2023
5e99a8f
PR responses
Feb 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .../Microsoft.Spark.Extensions.Delta.E2ETest/Microsoft.Spark.Extensions.Delta.E2ETest.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\Microsoft.Spark.E2ETest\Microsoft.Spark.E2ETest.csproj" />
<ProjectReference Include="..\..\Microsoft.Spark\Microsoft.Spark.csproj" />
<ProjectReference Include="..\Microsoft.Spark.Extensions.Delta\Microsoft.Spark.Extensions.Delta.csproj" />
</ItemGroup>
</Project>
<ItemGroup>
<PackageReference Include="Microsoft.DotNet.UpgradeAssistant.Extensions.Default.Analyzers" Version="0.4.336902">
AFFogarty marked this conversation as resolved.
Show resolved Hide resolved
suhsteve marked this conversation as resolved.
Show resolved Hide resolved
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>
</Project>
11 changes: 7 additions & 4 deletions ....Spark.Extensions.Hyperspace.E2ETest/Microsoft.Spark.Extensions.Hyperspace.E2ETest.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\Microsoft.Spark.E2ETest\Microsoft.Spark.E2ETest.csproj" />
<ProjectReference Include="..\..\Microsoft.Spark\Microsoft.Spark.csproj" />
<ProjectReference Include="..\Microsoft.Spark.Extensions.Hyperspace\Microsoft.Spark.Extensions.Hyperspace.csproj" />
</ItemGroup>
</Project>
<ItemGroup>
<PackageReference Include="Microsoft.DotNet.UpgradeAssistant.Extensions.Default.Analyzers" Version="0.4.336902">
suhsteve marked this conversation as resolved.
Show resolved Hide resolved
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>
</Project>
15 changes: 7 additions & 8 deletions src/csharp/Microsoft.Spark.E2ETest/Microsoft.Spark.E2ETest.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
<Content Include="Resources\*">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>

<ItemGroup>
<InternalsVisibleTo Include="Microsoft.Spark.Extensions.Delta.E2ETest" />
<InternalsVisibleTo Include="Microsoft.Spark.Extensions.Hyperspace.E2ETest" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Microsoft.Spark.E2ETest.ExternalLibrary\Microsoft.Spark.E2ETest.ExternalLibrary.csproj" />
<ProjectReference Include="..\Microsoft.Spark.Worker\Microsoft.Spark.Worker.csproj" />
<ProjectReference Include="..\Microsoft.Spark\Microsoft.Spark.csproj" />
</ItemGroup>

<ItemGroup>
<Compile Include="..\Microsoft.Spark.UnitTest\TestUtils\ArrowTestUtils.cs" />
<Compile Include="..\Microsoft.Spark.UnitTest\TestUtils\TemporaryDirectory.cs" />
</ItemGroup>

</Project>
<ItemGroup>
<PackageReference Include="Microsoft.DotNet.UpgradeAssistant.Extensions.Default.Analyzers" Version="0.4.336902">
suhsteve marked this conversation as resolved.
Show resolved Hide resolved
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>
</Project>
4 changes: 4 additions & 0 deletions src/csharp/Microsoft.Spark.Worker.UnitTest/CommandExecutorTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1059,7 +1059,9 @@ public void TestRDDCommandExecutor(Version sparkVersion, IpcOptions ipcOptions)
foreach (int input in inputs)
{
memoryStream.Position = 0;
#pragma warning disable SYSLIB0011 // Type or member is obsolete
formatter.Serialize(memoryStream, input);
#pragma warning restore SYSLIB0011 // Type or member is obsolete
values.Add(memoryStream.ToArray());
}

Expand Down Expand Up @@ -1089,9 +1091,11 @@ public void TestRDDCommandExecutor(Version sparkVersion, IpcOptions ipcOptions)
for (int i = 0; i < inputs.Length; ++i)
{
Assert.True(SerDe.ReadInt32(outputStream) > 0);
#pragma warning disable SYSLIB0011 // Type or member is obsolete
Assert.Equal(
mapUdf(i),
formatter.Deserialize(outputStream));
#pragma warning restore SYSLIB0011 // Type or member is obsolete
}

// Validate all the data on the stream is read.
Expand Down
12 changes: 5 additions & 7 deletions src/csharp/Microsoft.Spark.Worker.UnitTest/Microsoft.Spark.Worker.UnitTest.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Moq" Version="4.10.0" />
<PackageReference Include="Microsoft.DotNet.UpgradeAssistant.Extensions.Default.Analyzers" Version="0.4.336902">
suhsteve marked this conversation as resolved.
Show resolved Hide resolved
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Microsoft.Spark.Worker\Microsoft.Spark.Worker.csproj" />
<ProjectReference Include="..\Microsoft.Spark\Microsoft.Spark.csproj" />
</ItemGroup>

<ItemGroup>
<Compile Include="..\Microsoft.Spark.UnitTest\SparkFixture.cs" />
<Compile Include="..\Microsoft.Spark.UnitTest\TestUtils\ArrowTestUtils.cs" />
<Compile Include="..\Microsoft.Spark.UnitTest\TestUtils\TemporaryDirectory.cs" />
</ItemGroup>

</Project>
</Project>
2 changes: 2 additions & 0 deletions src/csharp/Microsoft.Spark.Worker/Command/RDDCommandExecutor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,9 @@ private void Serialize(
{
case CommandSerDe.SerializedMode.Byte:
BinaryFormatter formatter = s_binaryFormatter ??= new BinaryFormatter();
#pragma warning disable SYSLIB0011 // Type or member is obsolete
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry about the continued use of BinaryFormatter due to https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide. pragma'ing away is necessary to compile...but I worry this needs to be fixed soon.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we will need to fix this in upcoming PRs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah... I am worried about it, and we'll get hounded by component governance for this... but if it's a vulnerability, it's already a vulnerability now, so we may as well solve it while on .net 6

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1131 to defer

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it's just a MemoryStream this one is really easy to fix. BinaryFormatter doesn't give any value over just writing the length and bytes.

formatter.Serialize(stream, message);
#pragma warning restore SYSLIB0011 // Type or member is obsolete
break;
case CommandSerDe.SerializedMode.None:
case CommandSerDe.SerializedMode.String:
Expand Down
13 changes: 5 additions & 8 deletions src/csharp/Microsoft.Spark.Worker/Microsoft.Spark.Worker.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,25 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFrameworks>net461;netcoreapp3.1</TargetFrameworks>
<TargetFrameworks>net6.0</TargetFrameworks>
AFFogarty marked this conversation as resolved.
Show resolved Hide resolved
<TargetFrameworks Condition="'$(OS)' != 'Windows_NT'">netcoreapp3.1</TargetFrameworks>
<RootNamespace>Microsoft.Spark.Worker</RootNamespace>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
</PropertyGroup>

<ItemGroup>
<InternalsVisibleTo Include="Microsoft.Spark.Worker.UnitTest" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.DotNet.DependencyManager" Version="10.10.0-beta.20254.4" />
<PackageReference Include="System.Memory" Version="4.5.3" />
<PackageReference Include="Microsoft.DotNet.UpgradeAssistant.Extensions.Default.Analyzers" Version="0.4.336902">
suhsteve marked this conversation as resolved.
Show resolved Hide resolved
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>

<ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp3.1' ">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this isn't required in net 6?

<PackageReference Include="System.Runtime.Loader" Version="4.3.0" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Microsoft.Spark\Microsoft.Spark.csproj" />
</ItemGroup>

</Project>
</Project>
4 changes: 4 additions & 0 deletions src/csharp/Microsoft.Spark.Worker/Processor/BroadcastVariableProcessor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,15 +62,19 @@ internal BroadcastVariables Process(Stream stream)
$"server {readBid} is different from the Broadcast Id received " +
$"from the payload {bid}.");
}
#pragma warning disable SYSLIB0011 // Type or member is obsolete
object value = formatter.Deserialize(socket.InputStream);
#pragma warning restore SYSLIB0011 // Type or member is obsolete
BroadcastRegistry.Add(bid, value);
}
else
{
string path = SerDe.ReadString(stream);
using FileStream fStream =
File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read);
#pragma warning disable SYSLIB0011 // Type or member is obsolete
object value = formatter.Deserialize(fStream);
#pragma warning restore SYSLIB0011 // Type or member is obsolete
BroadcastRegistry.Add(bid, value);
}
}
Expand Down
Loading