ArPow stage 1: local source-build infrastructure #53294

MichaelSimons · 2021-05-26T15:50:49Z

This PR adds the local build infrastructure that lets ArPow (arcade-powered source-build) run in this repo. See https://github.com/dotnet/source-build/blob/master/Documentation/planning/arcade-powered-source-build/onboarding/local-onboarding.md for more details about how it works.

To try it out locally, run this on Linux: ./build.sh -c Release --restore --build --pack /p:ArcadeBuildFromSource=true -bl

The patches included in this PR were migrated from 5.0 source-build with any necessary updates for 6.0.

This PR should have no effect on ordinary builds, or CI. ArPow stage 2 will add source-build to CI: PR validation and official builds.

For https://github.com/dotnet/source-build/blob/master/Documentation/planning/arcade-powered-source-build/implementation-plan.md

Fixes: dotnet/source-build#2052

…and non-portable builds" This reverts commit 464010d.

…tage1 Conflicts: eng/Version.Details.xml src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_ssl.c

…tage1 Conflicts: eng/Version.Details.xml

MichaelSimons · 2021-06-14T21:31:32Z

src/libraries/libraries-packages.proj

                      Exclude="$(MSBuildThisFileDirectory)*\src\**\*.shproj"/>
  </ItemGroup>

  <!-- Need the PackageIndexFile file property from baseline.props -->
  <Import Project="$(PkgDir)baseline\baseline.props" />

  <!-- Calculate PackageDownload items for restore. -->
-  <Import Project="$(RepositoryEngineeringDir)restore\harvestPackages.targets" Condition="'$(MSBuildRestoreSessionId)' != ''" />
+  <Import Project="$(RepositoryEngineeringDir)restore\harvestPackages.targets" Condition="'$(MSBuildRestoreSessionId)' != '' and '$(DotNetBuildFromSource)' != 'true'" />


@ViktorHofer - I see you removed the 0027-Remove-libraries-packages.proj-harvesting.patch with 6c2e477#diff-a195cee0207dfe976258d38145c8edb053bbe098930d1b701b36aaba78aa9855. This breaks source-build with:

/repos/runtime/artifacts/source-build/self/src/eng/restore/harvestPackages.targets(12,5): error MSB4062: The "GetLastStablePackage" task could not be loaded from the assembly /repos/runtime/artifacts/source-build/self/package-cache/microsoft.dotnet.build.tasks.packaging/6.0.0-beta.21311.3/tools/net5.0/Microsoft.DotNet.Build.Tasks.Packaging.dll. Could not load file or assembly '/repos/runtime/artifacts/source-build/self/package-cache/microsoft.dotnet.build.tasks.packaging/6.0.0-beta.21311.3/tools/net5.0/Microsoft.DotNet.Build.Tasks.Packaging.dll'. The system cannot find the path specified. [/repos/runtime/artifacts/source-build/self/src/Build.proj] /repos/runtime/artifacts/source-build/self/src/eng/restore/harvestPackages.targets(12,5): error MSB4062: Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [/repos/runtime/artifacts/source-build/self/src/Build.proj]

I added the changes in the patch to fix source-build. What were you expecting to be the patch resolution?

I just inlined the patch into main an hour ago:

runtime/src/libraries/libraries-packages.proj

Line 25 in 6474c72

<Import Project="$(RepositoryEngineeringDir)restore\harvestPackages.targets" Condition="'$(MSBuildRestoreSessionId)' != '' and '$(DotNetBuildFromSource)' != 'true'" />

…tage1 Conflicts: eng/Version.Details.xml src/libraries/libraries-packages.proj

MichaelSimons · 2021-06-15T14:30:56Z

eng/source-build-patches/0036-Exclude-ReadyToRun.Experimental-NuGet-package.patch

+     <ProjectReference Include="Microsoft.NETCore.ILDAsm\Microsoft.NETCore.ILDAsm.proj" />
+  </ItemGroup>
+
+  <ItemGroup Condition="'$(DotNetBuildFromSource)' != 'true'">


Integrating this patch with #54222.

MichaelSimons · 2021-06-15T14:32:17Z

eng/source-build-patches/0023-Remove-RuntimeIdentifiers-tag-to-prevent-aspnet-pack.patch

+     <OutputPath>$(RuntimeBinDir)/dotnet-pgo</OutputPath>
+     <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
+     <EnableDefaultEmbeddedResourceItems>false</EnableDefaultEmbeddedResourceItems>
+-    <RuntimeIdentifiers>linux-x64;linux-musl-x64;win-x64</RuntimeIdentifiers>


After discussing this with @dseefeld, we believe this patch is obsolete because dotnet-pgo.csproj is now excluded when building source-build.

…tage1 Conflicts: eng/Version.Details.xml

eng/source-build-patches/0019-Fix-missing-crossgen2-for-non-portable-RID.patch

MichaelSimons · 2021-06-17T13:39:20Z

eng/SourceBuild.props

+  <Target Name="CategorizeRuntimeSupplementalArtifacts"
+          BeforeTargets="GetCategorizedIntermediateNupkgContents">
+    <ItemGroup>
+      <!--


FYI - I logged an issue for having the ArPOW infrastructure handle large packages.

…tage1

MichaelSimons · 2021-06-18T21:07:53Z

Reviewers, please take another look and provide feedback. There are only 3 patches remaining. What needs to be resolved before we can merge? Thanks

ericstj

I'm not seeing anything here that's blocking, but I'd like other reviewers to weigh in too. I made a couple nice to have suggestions.

ViktorHofer

This looks really great now. Thanks for all your time and dedication @MichaelSimons 💯

MichaelSimons · 2021-06-22T01:16:45Z

Looks like the (CoreCLR Pri0 Runtime Tests Run windows arm64 checked) failure is happening for other PRs as well on the same test case (.packages\microsoft.dotnet.helix.sdk\6.0.0-beta.21311.3\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets#L78)- e.g. https://github.com/dotnet/runtime/pull/54522/checks?check_run_id=2879806663

dagood and others added 30 commits February 3, 2021 23:57

Initial arcade-powered source-build infra

1054536

Add patches, fixup from 5.0

7d0c956

Rename patches to match 6.0-p1 work

376c10c

Add source-build specific build script

8a4c258

Incorporate build-source-build.sh, into eng/

9d474ce

Run inner build script, through coreclr

06f1d5e

Initial source-build args based on live build (not props file)

0e4d155

Cleanup: add RID comments, rm empty/absolute args

cc4124f

Fix subsets

59fb50e

Disable sourcelink in env, not args

ad4201b

Fix CI NZEC

1c87b39

Revert "Use same code to DetectCiphersuiteConfiguration for portable …

c4f711d

…and non-portable builds" This reverts commit 464010d.

Fix portability

479c95a

Fix AllJits build

2565431

Fix missing crossgen2 for non-portable RID

cd313cd

Create supplemental intermediate nupkgs

2b5307d

Tweak category names

d7546c5

Use centralized supplemental nupkg infra

d6602eb

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

843caa7

…tage1 Conflicts: eng/Version.Details.xml src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_ssl.c

Add additional 6.0 patches

a7d8b95

Patch updates after merging in main

18d9616

SourceBuild.props cleanup

f8e8475

Fix issue with incorrect patch merge

aa13d3c

Patch updates

38cea15

Edit clr.tools patch

f5445ee

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

4d94fd4

…tage1 Conflicts: eng/Version.Details.xml

patch updates

0a8059d

Revert patch integration

9b2c5b3

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

1b2d9b1

…tage1 Conflicts: eng/Version.Details.xml

Patch updates

23e4735

MichaelSimons added 3 commits June 14, 2021 20:31

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

1c97495

…tage1 Conflicts: eng/Version.Details.xml

Patch updates necessary with latest main merge

e6f5d8f

Add back libraries-packages.proj harvesting patch

f026627

MichaelSimons commented Jun 14, 2021

View reviewed changes

MichaelSimons added 4 commits June 14, 2021 22:34

Refactor intermediate package split to be chunkier

371a532

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

58bb935

…tage1 Conflicts: eng/Version.Details.xml src/libraries/libraries-packages.proj

Integrate patch 0017

309421d

Subsets update per code review

7230fa1

MichaelSimons commented Jun 15, 2021

View reviewed changes

MichaelSimons added 2 commits June 15, 2021 14:36

Remove obsolete patch

972035a

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

f905b0e

…tage1 Conflicts: eng/Version.Details.xml

MichaelSimons commented Jun 15, 2021

View reviewed changes

eng/source-build-patches/0019-Fix-missing-crossgen2-for-non-portable-RID.patch Outdated Show resolved Hide resolved

karelz mentioned this pull request Jun 17, 2021

[wasm] SendReceive_VaryingLengthBuffers_Success failing in CI #53957

Closed

MichaelSimons commented Jun 17, 2021

View reviewed changes

MichaelSimons added 2 commits June 18, 2021 14:58

Merge branch 'main' of https://github.com/dotnet/runtime into arpow-s…

0124b4f

…tage1

Removed patches that were integrated into main

cf48c62

ericstj approved these changes Jun 21, 2021

View reviewed changes

ViktorHofer added 2 commits June 21, 2021 19:35

Remove two additional patches

4211c03

Remove remaining patches

88a1715

ViktorHofer approved these changes Jun 21, 2021

View reviewed changes

dseefeld approved these changes Jun 21, 2021

View reviewed changes

hoyosjs approved these changes Jun 21, 2021

View reviewed changes

crummel approved these changes Jun 22, 2021

View reviewed changes

ViktorHofer merged commit 849033a into dotnet:main Jun 22, 2021

MichaelSimons deleted the arpow-stage1 branch June 22, 2021 13:23

ghost locked as resolved and limited conversation to collaborators Jul 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ArPow stage 1: local source-build infrastructure #53294

ArPow stage 1: local source-build infrastructure #53294

MichaelSimons commented May 26, 2021 •

edited

Loading

MichaelSimons Jun 14, 2021

ViktorHofer Jun 14, 2021 •

edited

Loading

MichaelSimons Jun 15, 2021

MichaelSimons Jun 15, 2021

MichaelSimons Jun 17, 2021

MichaelSimons commented Jun 18, 2021

ericstj left a comment

ViktorHofer left a comment

MichaelSimons commented Jun 22, 2021 •

edited

Loading

ArPow stage 1: local source-build infrastructure #53294

ArPow stage 1: local source-build infrastructure #53294

Conversation

MichaelSimons commented May 26, 2021 • edited Loading

MichaelSimons Jun 14, 2021

Choose a reason for hiding this comment

ViktorHofer Jun 14, 2021 • edited Loading

Choose a reason for hiding this comment

MichaelSimons Jun 15, 2021

Choose a reason for hiding this comment

MichaelSimons Jun 15, 2021

Choose a reason for hiding this comment

MichaelSimons Jun 17, 2021

Choose a reason for hiding this comment

MichaelSimons commented Jun 18, 2021

ericstj left a comment

Choose a reason for hiding this comment

ViktorHofer left a comment

Choose a reason for hiding this comment

MichaelSimons commented Jun 22, 2021 • edited Loading

MichaelSimons commented May 26, 2021 •

edited

Loading

ViktorHofer Jun 14, 2021 •

edited

Loading

MichaelSimons commented Jun 22, 2021 •

edited

Loading