Build emsdk from source

[directhex]

Previously, our emsdk packages have effectively just wrapped the emsdk binaries downloaded by the emsdk.py script in the repository, built by upstream at some time and shipped out to users. This PR replaces all those pieces with files built on Microsoft infrastructure and shipped in via transport packages from those repositories. As part of this, we are no longer constrained by the architecture limitations of upstream and can ship native ARM64 builds for Mac, Linux and Windows.

By uncoupling ourselves from upstream, it enables us to diverge from them on how we build things, which versions are included, etc. As of now, the headline differences are:

• Download size reduced by ~40% (linux-x64 result, may vary by platform)
• Binaryen tools built with dynamic linking instead of static (this is a major source of space savings)
• Native osx-arm64 and win-arm64 builds replace the previous redirects to x64 emulation, and a new linux-arm64 build has been added to the workload

Note that downloaded components are not 101% eliminated - Python is built using binary versions of OpenSSL which we are not building ourselves. This can be resolved later.

Closes: dotnet/runtime#75613

[radical]

Just for reference, if it helps: https://learn.microsoft.com/en-us/visualstudio/msbuild/property-functions?view=vs-2022#msbuild-ensuretrailingslash

[radical]

Just for future reference really: $([MSBuild]::NormalizeDirectory(..)) - this will use the correct slash, and ensure the trailing slash.

[radical]

As a potential follow up, it would be useful to run WBT from dotnet/runtime on PRs here.

1. get the emsdk packages from here
2. build runtime with /p:MicrosoftNETRuntimeEmscriptenVersion=<version from step 1>
3. and run wasm.build.tests (see eng/pipelines/common/templates/wasm-build-tests.yml in dotnet/runtime)

I think this should be enough.

[directhex]

I was planning on throwing this into General Testing in the morning, to do a test PR in runtime

[radical]

Btw I meant adding a CI job here.

[directhex]

hopefully https://github.com/dotnet/runtime/pull/88687/checks runs and does something meaningful

[directhex]

FFS, the required test branch was skipped. What do I have to do to get wasm-build-tests.yml to run in CI?

[radical]

FFS, the required test branch was skipped. What do I have to do to get wasm-build-tests.yml to run in CI?

What do you need? Run just for testing? It is run based on path changes. but you can force it for testing if you want.

      - template: /eng/pipelines/common/templates/wasm-build-tests.yml
        parameters:
          platforms:
            - browser_wasm
            - browser_wasm_win
          alwaysRun: ${{ variables.isRollingBuild }}
          extraBuildArgs: /p:AotHostArchitecture=x64 /p:AotHostOS=$(_hostedOS)

.. in runtime.yml set alwaysRun: true.

[radical]

Also, runtime-wasm pipeline will skip any jobs that would be running on runtime.

[radical]

It's running on wbt - https://dev.azure.com/dnceng-public/public/_build/results?buildId=336983&view=logs&j=1fa93050-f528-55d3-a351-f8bf9ce5adbf

[directhex]

Okay. Good. Great! Looks like https://dev.azure.com/dnceng-public/public/_build/results?buildId=336983&view=logs&j=1fa93050-f528-55d3-a351-f8bf9ce5adbf&t=d9dc6d3d-3442-5b70-1296-5b62e29ea209 is the one we're talking about. And it's explicitly installing my manifest from General Testing:

- Installing manifest: Microsoft.NET.Workload.Emscripten.Current/8.0.0-preview.7.23361.1

So the results on Helix on here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=336983&view=logs&j=1fa93050-f528-55d3-a351-f8bf9ce5adbf&t=a9f1a437-3b59-5900-1137-ec6cae9a530c will be using my self-built emsdk? That'll be it?

[radical]

Awesome 🎆 🚀