Container offering for supporting Native AOT scenarios

[mthalman]

We need to determine if and how we should adjust the set of container images we provide to meet the needs of Native AOT scenarios. It looks like the set of package dependencies that are needed for those scenarios is different from standard deployments.

For example, there is work to remove a dependency on libstdc++: dotnet/runtime#76655. That likely has no impact on the current full versions of Debian, Ubuntu, and Mariner since that package is installed by default. But it would impact Alpine, distroless Mariner, and chiseled Ubuntu since it's explicitly being added in those images.

We should avoid the inclusion of any unnecessary packages in our images. It leads to a larger image size and increased exposure to vulnerabilities.

[mthalman]

If there were to be separate images only intended for Native AOT deployments, this would be limited to just runtime-deps images because it is, by definition, a trimmed application that has the runtime embedded in it.

[eerhardt]

Also consider containers that support building NativeAOT apps.

Right now:

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
RUN apt update
RUN apt install -y clang zlib1g-dev

Is needed in order to publish with PublishAot=true.

[mthalman]

[Triage]
First step is to investigate the scenario with an app building exercise. This will help to determine what the requirements are for the different Linux distros that are relevant. At the very least, we could then provide sample documentation to illustrate how to configure Dockerfiles for this scenario. We can then make a determination of whether it's worthwhile to actually provide official base images.

[mthalman]

One interesting aspect of native AOT publishing is the following from https://learn.microsoft.com/dotnet/core/deploying/native-aot/:

A native AOT binary produced on Linux machine is only going to work on same or newer Linux version. For example, native AOT binary produced on Ubuntu 20.04 is going to run on Ubuntu 20.04 and later, but it is not going to run on Ubuntu 18.04.

This could have implications on which OS versions we make the SDK available on. For 7.0, for example, we only provide Ubuntu 22.04 images.

[xlievo]

sdk:
FROM mcr.microsoft.com/dotnet/sdk:7.0.103-alpine3.17-amd64 AS build
RUN apk update && apk upgrade
RUN apk add --no-cache clang build-base zlib-dev libssl1.1

runtime:
FROM alpine:3.17
RUN apk update && apk upgrade
RUN apk add --no-cache libstdc++

PublishAot=true

[eerhardt]

RUN apk add --no-cache libstdc++

Note that in .NET 8, libstdc++ is no longer needed. This was fixed with dotnet/runtime#79501.

[mthalman]

[Triage]
Barring any extraordinary circumstances, a goal should be to have a single runtime-deps image that can be used for both normal deployments as well as NativeAOT. We should avoid having a special runtime-deps image to satisfy NativeAOT scenarios.

The question then is around the SDK and what its requirements are. Is it possible to have a single SDK image to satisfy all scenarios for that as well?

[jkotas]

a goal should be to have a single runtime-deps image that can be used for both normal deployments as well as NativeAOT

The current runtime-deps image works for NativeAOT deployments already. It is bigger than strictly necessary, but that is not the end of the world. If somebody wants the absolute minimal footprint, they can build their own.

The question then is around the SDK and what its requirements are.

The requirements are documented in https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/#prerequisites . I do not think it is a good idea to install these prerequisites to SDK image used everywhere. They have non-trivial size.

[richlander]

After some evaluation of this scenario, I've come to the conclusion that we should do something here. There was initial concern about this because the naive (and not unreasonable) reaction is that we're adding a whole other pivot to our container image catalog.

I propose that we provide an excellent experience for Native AOT for one of our support Linux variants, specifically Ubuntu. Ubuntu is a great choice for two reasons: glibc compatible, and Chiseled.

We'd offer two images (for x64 and Arm64; Arm32 is debatable):

• Ubuntu SDK w/clang and zlib1g-dev installed.
• Ubuntu Chiseled image with the minimal requirements (no libstdc++).

Perhaps crazy, but we could just install these two components in the Ubuntu SDK image and just call it good. For folks that are unhappy with the size increase, they could use the Debian image. Clearly, we'd need to see the actual size increase to decide if this is OK.

If Chiseled is a step too far for some folks, they can go DIY (which is status quo).

[ivanjx]

currently i cant build for arm64 on amd64 with docker. always fails at restoring nuget packages:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

i tried to run dotnet restore separately but i got another weird errors:

33.47 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(314,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/build/project.csproj]
61.91 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(378,5): error MSB4018: The "DefineStaticWebAssets" task failed unexpectedly. [/build/project.csproj]
61.91 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(378,5): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/build/project.csproj]
61.91 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(378,5): error MSB4018:    at InvokeStub_DefineStaticWebAssets.set_CandidateAssets(Object, Object, IntPtr*) [/build/project.csproj]
61.91 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(378,5): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/build/project.csproj]
61.92 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(379,7): error MSB4026: The "CandidateAssets=@(_PackageProjectBundleCandidates)" parameter for the "DefineStaticWebAssets" task is invalid. [/build/project.csproj]
61.92 /usr/share/dotnet/sdk/8.0.100-preview.6.23330.14/Sdks/Microsoft.NET.Sdk.StaticWebAssets/targets/Microsoft.NET.Sdk.StaticWebAssets.ScopedCss.targets(378,5): error MSB4063: The "DefineStaticWebAssets" task could not be initialized with its input parameters.  [/build/project.csproj]

[ivanjx]

here is what i tried building arm64 with amd64 docker image (since apparently sdk does not support qemu):

FROM mcr.microsoft.com/dotnet/sdk:8.0.100-preview.6-bookworm-slim-amd64 as build
RUN dpkg --add-architecture arm64 && \
    apt-get update
RUN apt-get install -y \
    clang zlib1g-dev:arm64 \
    binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu
COPY ./project /build
WORKDIR /build
RUN dotnet publish \
    -c Release \
    -r linux-arm64 \
    -o /output

unfortunately fails at running objcopy:

objcopy: Unable to recognise the format of the input file `bin/Release/net8.0/linux-arm64/native/project'

EDIT:
adding these hacks after apt-get install works:

RUN rm /usr/bin/objcopy && \
    ln -s /usr/bin/aarch64-linux-gnu-objcopy /usr/bin/objcopy

[richlander]

Let's move the cross-targeting topic to dotnet/runtime#88971

[MichaelSimons]

I propose that we provide an excellent experience for Native AOT for one of our support Linux variants, specifically Ubuntu.

@richlander - what is the reasoning/thinking for supporting one variant versus a complete matrix? I can draw some conclusions but would like to hear yours.

[richlander]

I did a bit of digging to provide more informed insight.

$ docker images mcr.microsoft.com/dotnet/sdk 
REPOSITORY                     TAG                 IMAGE ID       CREATED      SIZE
mcr.microsoft.com/dotnet/sdk   8.0-preview-jammy   174be2ee75d7   6 days ago   818MB
$ docker images sdk-8.0-preview-jammy-clang 
REPOSITORY                    TAG       IMAGE ID       CREATED          SIZE
sdk-8.0-preview-jammy-clang   latest    5330d6b29eed   28 seconds ago   1.63GB
$ cat Dockerfile.ubuntu 
FROM mcr.microsoft.com/dotnet/sdk:8.0-preview-jammy

RUN apt-get update && apt-get -y install clang zlib1g-dev

That obviously means that we cannot add these packages to an existing SDK.

My thinking is that we should do the following:

• Produce a new aot-deps flavor of images, targeted at the distroless distro flavors we support.
• Produce a new aot-sdk flavor of images for those same distros, with the same basic layering as my Dockerfile demonstrates.
• Everything else is DIY.

That approach is coherent, delivers significant value, and is focused on E2E workflows.

Names of the repos TBD. Those are provided as placeholders.

[richlander]

I've now played with Dockerfiles that support native AOT a lot more as part of #4742. I thought it would be useful to share a vision for what the user Dockerfile would look like.

FROM mcr.microsoft.com/dotnet/aot-sdk:8.0-jammy AS build
ARG TARGETARCH
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore -a $TARGETARCH

# copy and publish app and libraries
COPY . .
RUN dotnet publish -a $TARGETARCH --no-restore -o /app releasesapi.csproj
RUN rm /app/*.dbg /app/*.Development.json


# final stage/image
FROM mcr.microsoft.com/dotnet/aot-deps:8.0-jammy-chiseled
WORKDIR /app
COPY --from=build /app .
USER $APP_UID
ENTRYPOINT ["./releasesapi"]

Close watchers will notice that this Dockerfile removes almost everything that makes a native AOT Dockerfile special leaving it to look remarkably similar to a regular CoreCLR app. Exactly. The assumption is that the project file has all the properties that specify native AOT.

This design relies on manifest tags. There is a design choice about whether to make the aot-sdk enable enable cross-compilation (by architecture) by default. We should do that if (A) the size increase isn't onerous and the use of multiple architecture feeds doesn't cause problems.

This is the size difference:

[rich@kelowna releasesapi-aot]$ docker images ubuntu-sdk
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ubuntu-sdk   latest    7a1192602288   4 hours ago   1.8GB
[rich@kelowna releasesapi-aot]$ docker images ubuntu-cross-sdk
REPOSITORY         TAG       IMAGE ID       CREATED       SIZE
ubuntu-cross-sdk   latest    cdfbb3b521cb   6 hours ago   1.88GB

They are almost identical. So, size doesn't seem to be an important metric. I was originally proposing aot-sdk and aot-cross-sdk. That doesn't seem to be warranted.

That means that users could docker build that Dockerfile on x64 or Arm64 and use the --platform switch to cross-compile. Sounds nice!

[lbussell]

The current runtime-deps image works for NativeAOT deployments already. It is bigger than strictly necessary, but that is not the end of the world. If somebody wants the absolute minimal footprint, they can build their own.

@jkotas, what are the absolute minimum requirements for native AOT deployments, are they documented somewhere? From my testing, I'm able to run native AOT ASP.NET apps directly on the base images - ubuntu.azurecr.io/ubuntu:jammy, mcr.microsoft.com/cbl-mariner/base/core:2.0, and alpine. It seems like it only makes sense to offer aot-deps images if they were distroless.

[richlander]

Answered here: #4748 (comment)

[jetersen]

This could also benefit the -p:PublishProfile=DefaultContainer together with the PublishAot property so that the output would be an image that was of the rootless, distroless flavor? 🤔

[lbussell]

Images are available in mcr.microsoft.com/dotnet/nightly complete with the testing in #4859

[lbussell]

This could also benefit the -p:PublishProfile=DefaultContainer together with the PublishAot property so that the output would be an image that was of the rootless, distroless flavor? 🤔

Maybe @baronfel can comment on support for this?

[baronfel]

That would be an ideal next step - we already look at the presence of SelfContained to see if we should use runtime-deps, and this would be a continuation of that. These are only on nightly, right? We don't have a good story yet for how a user would signal that they were ok with nightly-level images. At least until we have that, users will need to be explicit and use ContainerBaseImage to point to these new images.

[devnoname120]

Cross-posting my comment: #4859 (comment)

It has been available in nightly for a year already. Could you consider merging it to stable? Thanks!

[lbussell]

@devnoname120 We are keeping a close eye on the pull stats for these images. At the moment, there's not enough usage of the preview images to warrant publishing so many new images.

I left a comment on your linked PR - you can get really close to the same thing by installing the additional packages needed for AOT compilation in your SDK image. And if you wanted to use Ubuntu instead of Debian, for example, you could check out the individual Dockerfile that we publish for the nightly AOT SDK images (it also contains some extra packages for cross-compilation support - not all packages are strictly necessary) https://github.com/dotnet/dotnet-docker/blob/nightly/src/sdk/8.0/jammy-aot/amd64/Dockerfile.

Also related: #5020