Login problem with Private Registry

[anikolop]

Having problem pushing to private registry.

Using 7.0.200-preview.22628.1

Project Setup:

<PublishProfile>DefaultContainer</PublishProfile>
<ContainerImageName>demoapi</ContainerImageName>
<ContainerImageTags>0.0.1-alpha;latest</ContainerImageTags>
<ContainerRegistry>myregistry.example.com</ContainerRegistry>

Publishing with cli:

dotnet publish --os linux --arch arm64 -c Release

Publish Output:

MSBuild version 17.5.0-preview-22620-02+a6f6699d1 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
C:\Program Files\dotnet\sdk\7.0.200-preview.22628.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  Demo7.Application.Extensions -> C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Application.Extensions\bin\Release\net7.0\Demo7.Application.Extensions.dll
  Demo7.Api -> C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\bin\Release\net7.0\linux-arm64\Demo7.Api.dll
  Demo7.Api -> C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\bin\Release\net7.0\linux-arm64\publish\
  Building image 'demoapi' with tags 0.0.1-alpha,latest on top of base image mcr.microsoft.com/dotnet/aspnet:7.0
  Uploading layer sha256:934ce60d1040c5d4922bae5879321a398777457b7514de02ef69ece49e6aa907 to myregistry.example.com
  Uploading layer sha256:58419453adb26cc29a2cfa8879b86dd435d7b858acead3c19ca1d9de5ba1cf71 to myregistry.example.com
  Uploading layer sha256:f62b34a82112da4899d077a084b67d0b4554be93b3a90a29c5b2580f233c5b26 to myregistry.example.com
  Uploading layer sha256:53a2b5ccd9af9fa9875417f9c8566374f3ffc5dd463b2993fccc357d60811255 to myregistry.example.com
  Uploading layer sha256:f644f71d54a995dd2435f6ba15c28e88a18a12a1f55bea2e28e22281d6158ebe to myregistry.example.com
  Uploading layer sha256:11e7d236aff075f6bc3e14b38e26760e2394d6183a01e55becc4e02e05fc375f to myregistry.example.com
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : Failed to push to the output registry: System.AggregateException: One or more errors occurred. (Failed to upload blob to https://myregistry.example.com/v2/demoapi/blobs/uploads/; received Unauthorized with detail {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"demoapi","Action":"pull"},{"Type":"repository","Class":"","Name":"demoapi","Action":"push"}]}]} [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : ) [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :  ---> System.ApplicationException: Failed to upload blob to https://myregistry.example.com/v2/demoapi/blobs/uploads/; received Unauthorized with detail {"errors":[{"code":"UNAUTHORIZED" ,"message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"demoapi","Action":"pull"},{"Type":"repository","Class":"","Name":"demoapi","Action":"push"}]}]} [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :  [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Registry.StartUploadSession(String name, String digest, HttpClient client) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 333 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Registry.UploadBlob(String name, String digest, Stream contents) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 371 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Registry.Push(Layer layer, String name, Action`1 logProgressMessage) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 251 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass40_0.<<Push>b__0>d.MoveNext() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 449 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error : --- End of stack trace from previous location --- [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Registry.Push(Image x, String name, String tag, String baseName, Action`1 logProgressMessage) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\Registry.cs:line 460 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    --- End of inner exception stack trace --- [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at System.Threading.Tasks.Task.Wait() [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]
  C:\Users\anikolop\.nuget\packages\microsoft.net.build.containers\0.3.2\build\Microsoft.NET.Build.Containers.targets(114,9): error :    at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\CreateNewImage.cs:line 149 [C:\Users\anikolop\source\repos\anikolop\Playground\Demo7\Demo7.Api\Demo7.Api.csproj]

Docker config.json:

{
  "auths": {
    "myregistry.example.com": {}
  },
  "credsStore": "desktop"
}

Logs from the registry:

time="2023-01-26T09:03:25.005205457Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential" go.version=go1.16.15 http.request.host=myregistry.example.com http.request.id=bd57d8f8-484d-4ee7-935d-6762de3491b6 http.request.method=HEAD http.request.remoteaddr=192.168.30.56 http.request.uri="/v2/demoapi/blobs/sha256:58419453adb26cc29a2cfa8879b86dd435d7b858acead3c19ca1d9de5ba1cf71" http.request.useragent=".NET Container Library" vars.digest="sha256:58419453adb26cc29a2cfa8879b86dd435d7b858acead3c19ca1d9de5ba1cf71" vars.name=demoapi 

[baronfel]

we should be using the credentials from the desktop credential store (the precedence is credHelpers > credStore > auths) so I'd check real quick that you were actually logged in to that registry. can you do a docker login myregistry.example.com flow and make sure you're authenticated through the docker engine, then rerun the publish?

[anikolop]

Even through i am definitely logged in, when i try to login again i get:

Authenticating with existing credentials...
Login Succeeded

And on publish the same error.
As temporary workaround i publish the image localy then push it through the command line to the registry.

[anikolop]

Had some time to look into and debug the code and found that TryParseAuthenticationInfo returns always false for my registry.
The registry was setup 3 years ago according to the example on the official docker hub. https://docs.docker.com/registry/deploying/#native-basic-auth
Changing the TryParseAuthenticationInfo from this

    private static bool TryParseAuthenticationInfo(HttpResponseMessage msg, [NotNullWhen(true)] out string? scheme, [NotNullWhen(true)] out AuthInfo? authInfo)
    {
        authInfo = null;
        scheme = null;

        var authenticateHeader = msg.Headers.WwwAuthenticate;
        if (!authenticateHeader.Any())
        {
            return false;
        }

        AuthenticationHeaderValue header = authenticateHeader.First();
        if (header is { Scheme: "Bearer" or "Basic", Parameter: string bearerArgs })
        {
            scheme = header.Scheme;
            Dictionary<string, string> keyValues = new();
            foreach (Match match in BearerParameterSplitter().Matches(bearerArgs))
            {
                keyValues.Add(match.Groups["key"].Value, match.Groups["value"].Value);
            }

            if (keyValues.TryGetValue("realm", out string? realm) && keyValues.TryGetValue("service", out string? service))
            {
                string? scope = null;
                keyValues.TryGetValue("scope", out scope);
                authInfo = new AuthInfo(new Uri(realm), service, scope);
                return true;
            }
        }

        return false;
    }

to this:

    private static bool TryParseAuthenticationInfo(HttpResponseMessage msg, [NotNullWhen(true)] out string? scheme, [NotNullWhen(true)] out AuthInfo? authInfo)
    {
        authInfo = null;
        scheme = null;

        var authenticateHeader = msg.Headers.WwwAuthenticate;
        if (!authenticateHeader.Any())
        {
            return false;
        }

        AuthenticationHeaderValue header = authenticateHeader.First();
        if (header is { Scheme: "Bearer" or "Basic", Parameter: string bearerArgs })
        {
            scheme = header.Scheme;
            Dictionary<string, string> keyValues = new();
            foreach (Match match in BearerParameterSplitter().Matches(bearerArgs))
            {
                keyValues.Add(match.Groups["key"].Value, match.Groups["value"].Value);
            }

            if (keyValues.TryGetValue("realm", out string? realm) && keyValues.TryGetValue("service", out string? service))
            {
                string? scope = null;
                keyValues.TryGetValue("scope", out scope);
                authInfo = new AuthInfo(new Uri(realm), service, scope);
                return true;
            }
            else if (header.Scheme.Equals("Basic"))
            {
                if (string.IsNullOrWhiteSpace(realm) || !Uri.IsWellFormedUriString(realm, UriKind.Absolute))
                {
                    realm = $"{msg.RequestMessage!.RequestUri!.Scheme}://{msg.RequestMessage!.RequestUri!.Host}";
                }
                authInfo = new AuthInfo(new Uri(realm), string.Empty, null);
                return true;
            }
        }

        return false;
    }

the image is pushed to the registry.

[baronfel]

Ok, so what I'm understanding here is that there is some kind of conventional default for the realm if one is not provided - do you know if there are any docs on this default, or a way that we could set up a similar test harness to make sure we don't regress?

[anikolop]

I think the problem is that realm is not always necessary a url. Mozilla here has the following example for basic authentication
WWW-Authenticate: Basic realm="Access to the staging site", charset="UTF-8"
And the official docker documentation for hosting a registry with native authentication has the following example where Realm has the value "Registry Realm"

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v "$(pwd)"/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v "$(pwd)"/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

Also the registry, by default, does not return any service header in the response and so the if (keyValues.TryGetValue("realm", out string? realm) && keyValues.TryGetValue("service", out string? service)) is never true.

For a test setup you can run the command above but without the 3 lines for the certificates. Then on the client enable the insecure registry in docker daemon.json for mac or linux or in the desktop engine setting for windows.
And the documentation is here: https://docs.docker.com/registry/insecure/

[baronfel]

This is great - it should be relatively easy to capture this in a test thread and then make this change. I appreciate all the investigation you've done here!

[wazzamatazz]

Worth noting as well that the call to GetAuthenticationAsync only passes in the Uri.Host property as the registry name.

If the private registry is running on a non-standard port (e.g. 5000 as is used in Docker's own examples when using the registry image) and you call docker login myserver:5000, a credential gets added to the Windows credential store for network address myserver:5000. Based on the code I've linked to above, GetAuthenticationAsync would attempt to get credentials for myserver instead of myserver:5000 and presumably fail to retrieve them.

Maybe the call to GetAuthenticationAsync should pass in $"{Uri.Host}:{Uri.Port}" as the registry name if the port does not match the default port for the URL scheme?

[baronfel]

Great call-out - I think we've surfaced this concept elsewhere in the code as the RegistryName, which should also be used to find the authentication information.

[avber]

Hi @baronfel ,

any update on this issue?

I'm also having troubles publishing containers. However, in my case I can adjust the www-authenticate header.

These values don't work.
www-authenticate: Basic realm="https://reg.mydomain.tld" service="registry"
www-authenticate: Basic realm="reg.mydomain.tld" service="registry"

Thanks

[avber]

@baronfel

Will this issue be fixed before code freeze on this repo?

Thanks

[baronfel]

Hey @avber sadly we won't be able to get to this before the code freeze, but we do want to spend the next iteration knocking down bugs like this.

[avber]

Hi @baronfel ,

Thanks for the info.

What values should I put into the www-authenticate header (see above) to make the current implementation work?

[baronfel]

Assigned this one to me because I have a PR out for it that is almost ready.

[baronfel]

This should be fixed in .NET 8.