BuildX for arm64 on .Net 7.0 is broken

[zimbres]

When try to build arm64 containers using .Net7 results in a error on restore step.

Just create a new web api for example with .Net7 as target framework and try to build it with buildx. For amd64 build works just fine.

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker buildx build --platform linux/arm64 -t zimbres/espupdater -f .\src\EspUpdater\Dockerfile --push . --no-cache
[+] Building 66.2s (12/17)
 => [internal] load .dockerignore                                                                                                                                                                                          0.1s
 => => transferring context: 382B                                                                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                       0.1s
 => => transferring dockerfile: 622B                                                                                                                                                                                       0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:7.0                                                                                                                                                          2.3s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:7.0                                                                                                                                                       2.0s
 => [build 1/7] FROM mcr.microsoft.com/dotnet/sdk:7.0@sha256:39069a126f2e3dd8f3aa72a21688d4de13ba23b051a8c5856be1460f8165c422                                                                                             52.0s
 => => resolve mcr.microsoft.com/dotnet/sdk:7.0@sha256:39069a126f2e3dd8f3aa72a21688d4de13ba23b051a8c5856be1460f8165c422                                                                                                    0.0s
 => => sha256:d508dcfcdd2e1834750cbcc43a9e0f44625d7cdbb349be5342f069a0a26a9d20 13.42MB / 13.42MB                                                                                                                           8.7s
 => => sha256:edd0ccac63956573208572413f089dcc51a922baced37a1c3588d55701aab5c3 150.47MB / 150.47MB                                                                                                                        38.8s
 => => sha256:62558e356c6cd900192712f446fefcd8029f9f54b12c87b4b342a392d2e9e9ed 25.39MB / 25.39MB                                                                                                                          11.1s
 => => extracting sha256:62558e356c6cd900192712f446fefcd8029f9f54b12c87b4b342a392d2e9e9ed                                                                                                                                  1.7s
 => => extracting sha256:edd0ccac63956573208572413f089dcc51a922baced37a1c3588d55701aab5c3                                                                                                                                  3.2s
 => => extracting sha256:d508dcfcdd2e1834750cbcc43a9e0f44625d7cdbb349be5342f069a0a26a9d20                                                                                                                                  0.3s
 => [internal] load build context                                                                                                                                                                                          0.1s
 => => transferring context: 6.72kB                                                                                                                                                                                        0.0s
 => [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:7.0@sha256:cc60f5590843f067871d72ffc99d3065f8556624d45465948166b11a08714ddc                                                                                           22.7s
 => => resolve mcr.microsoft.com/dotnet/aspnet:7.0@sha256:cc60f5590843f067871d72ffc99d3065f8556624d45465948166b11a08714ddc                                                                                                 0.0s
 => => sha256:9855365906865f18e48e4132950bb1f031a37afd9487af259eb8618313aaa672 9.80MB / 9.80MB                                                                                                                             6.2s
 => => sha256:e48426b90b5fe27894046a6b96e67e208ec25e2768a1c5bf76d233fa1ec1e7d6 155B / 155B                                                                                                                                 0.3s
 => => sha256:eafd3893bf918fb60399acce35d47fc92934372f519a0dc8ea56b7ae552a1524 30.71MB / 30.71MB                                                                                                                          19.1s
 => => sha256:dd6189d6fc13cb03db0f4a3d9659b6b6044fd5858019d659001eaf8367584d67 30.06MB / 30.06MB                                                                                                                          19.4s
 => => sha256:8487b5d2f3511cc5ae78d3f730d7f7e8f06466f6281464355fd6a464c95cab36 14.92MB / 14.92MB                                                                                                                           9.2s
 => => extracting sha256:dd6189d6fc13cb03db0f4a3d9659b6b6044fd5858019d659001eaf8367584d67                                                                                                                                  1.5s
 => => extracting sha256:8487b5d2f3511cc5ae78d3f730d7f7e8f06466f6281464355fd6a464c95cab36                                                                                                                                  0.4s
 => => extracting sha256:eafd3893bf918fb60399acce35d47fc92934372f519a0dc8ea56b7ae552a1524                                                                                                                                  0.8s
 => => extracting sha256:e48426b90b5fe27894046a6b96e67e208ec25e2768a1c5bf76d233fa1ec1e7d6                                                                                                                                  0.0s
 => => extracting sha256:9855365906865f18e48e4132950bb1f031a37afd9487af259eb8618313aaa672                                                                                                                                  0.3s
 => [base 2/2] WORKDIR /app                                                                                                                                                                                                0.7s
 => [final 1/2] WORKDIR /app                                                                                                                                                                                               0.1s
 => [build 2/7] WORKDIR /src                                                                                                                                                                                               1.4s
 => [build 3/7] COPY [src/EspUpdater/EspUpdater.csproj, src/EspUpdater/]                                                                                                                                                   0.0s
 => ERROR [build 4/7] RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"                                                                                                                                               10.5s
------
 > [build 4/7] RUN dotnet restore "src/EspUpdater/EspUpdater.csproj":
#0 10.13 /usr/share/dotnet/sdk/7.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(55,5): error MSB4184: The expression "[MSBuild]::GetTargetFrameworkVersion(net7.0, 2)" cannot be evaluated. Exception has been thrown by the target of an invocation. [/src/src/EspUpdater/EspUpdater.csproj]
------
Dockerfile:8
--------------------
   6 |     WORKDIR /src
   7 |     COPY ["src/EspUpdater/EspUpdater.csproj", "src/EspUpdater/"]
   8 | >>> RUN dotnet restore "src/EspUpdater/EspUpdater.csproj"
   9 |     COPY . .
  10 |     WORKDIR "/src/src/EspUpdater"
--------------------
ERROR: failed to solve: process "/bin/sh -c dotnet restore \"src/EspUpdater/EspUpdater.csproj\"" did not complete successfully: exit code: 1

Docker Version

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:28:44 2022
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.1 (90346)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:35 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Info

PS C:\Users\Marcio\OneDrive\Documents\VSCode\asp.net\EspUpdater> docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.5.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.3)
  extension: Manages Docker extensions (Docker Inc., v0.2.13)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 18
  Running: 4
  Paused: 0
  Stopped: 14
 Images: 40
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: BGP4:KFNX:A6XV:MDJT:AKET:PD7L:O7YG:6SHF:KNRB:EFFD:PD3W:6OC2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[toanxyz]

I have the same issue too, it’s blocking us right now.

[mthalman]

This appears to be a bug with NuGet running with QEMU (Quick Emulator). QEMU is being used here by Docker because you're running Arm64 on an x64 host machine.

In the meantime, let's talk about a workaround. You'll want to avoid emulation by not using a multi-arch tag for the sdk. To do this, you can use the 7.0-bullseye-slim-amd64 tag. You should only need to use that for the sdk stage of your Dockerfile. The final stage can still use the multi-arch tag (7.0).

But because you're explicitly using the amd64 version of the sdk image, you'll need to explicitly set the RID in the restore and publish commands. And because the RID is set you'll also need to disable the self-contained option if you don't want a self-contained deployment.

And there's one little gotcha with the architecture. In order to make it a truly multi-arch Dockerfile, you'd also need to account for when you specify --platform linux/amd64. In that case, the RID needs to use x64 not amd64, so there's some logic in the Dockerfile below which shows how to fix that up as well.

Note that in the Dockerfile below I make use of the TARGETARCH and TARGETOS arguments. You don't need to explicitly pass these arguments with the --build-arg option. They are automatically derived by using the --platform option. So you shouldn't need to change how you run the build command.

Example:

FROM mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim-amd64 AS build

ARG TARGETARCH
ARG TARGETOS

RUN arch=$TARGETARCH \
    && if [ "$arch" = "amd64" ]; then arch="x64"; fi \
    && echo $TARGETOS-$arch > /tmp/rid

WORKDIR /source

COPY *.csproj .
    
RUN dotnet restore -r $(cat /tmp/rid)

COPY . .
RUN dotnet publish -c Release -o /app -r $(cat /tmp/rid) --self-contained false --no-restore


FROM mcr.microsoft.com/dotnet/runtime:7.0
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mthalman]

I've just moved this to the sdk repo to have that team take a look.

[mthalman]

I've tracked this down to 7.0 Preview 3. In that version, this scenario was broken. It was working in Preview 2.

[marcpopMSFT]

@mthalman are you saying this has been broken since 7.0.100-preview3? I'm not sure I quite follow the scenario that's broken. Are you saying this is an amd64 SDK on arm64 or some other configuration?

From the error message, the error appears to be coming from an msbuild intrinsic function. @benvillalobos to take a look since this involves arm64. Ben, is there any way to get the exception that's thrown by msbuild here?
#0 10.13 /usr/share/dotnet/sdk/7.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(55,5): error MSB4184: The expression "[MSBuild]::GetTargetFrameworkVersion(net7.0, 2)" cannot be evaluated. Exception has been thrown by the target of an invocation. [/src/src/EspUpdater/EspUpdater.csproj]

[mthalman]

are you saying this has been broken since 7.0.100-preview3?

That's correct. And it was working in Preview 2. I verified this by changing the tag name used for the sdk image. Instead of the 7.0 image tag used by the original repro, I used 7.0.100-preview.2 and 7.0.100-preview.3. It worked on the former and broke on the latter.

I'm not sure I quite follow the scenario that's broken. Are you saying this is an amd64 SDK on arm64 or some other configuration?

This an Arm64 Docker container running on an x64 host machine. Docker uses QEMU to do this.

[toanxyz]

I have the issue with buildx saying:
The "CheckIfPackageReferenceShouldBeFrameworkReference" task's outputs could not be retrieved from the "ShouldAddFrameworkReference" parameter. Object reference not set to an instance of an object.

[benvillalobos]

At first glance I would pin this on the NuGet assembly. In this intrinsic function MSBuild reflects over NuGet.Frameworks.dll. I wonder if it's failing to call the Parse method within the assembly, possible because of the emulation or something changed in that dll?

I'm not sure if there's a way to get more info from the exception without debugging into the intrinsic method. @rainersigwald might know of some msbuild magic here.

[rainersigwald]

I don't see anything that would get better info: dotnet/msbuild#8151.

@benvillalobos you could try to repro with a private MSBuild that dumps the whole stack. That should be easier than trying to attach a debugger to the emulated ARM64 process inside the container.

[ghost]

Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!

If you believe this issue was closed out of error, please comment to let us know.

Happy Coding!

[mthalman]

I've created a new issue for this at: NuGet/Home#12227

[WoZ]

For everyone who found this ticket - read an explanation - https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support/