AddReference between containers

[simonhoss]

Hey

I'm trying to connect to container together with WithReference. But I'm not able to get the AllocatedEndpointAnnotation.

Here is a very simple example. (I'm aware that this would not make a lot of sense ;) )

var redis = builder.AddRedisContainer("redis");
var rabbitMq = builder.AddRabbitMQContainer("rabbitmq");

rabbitMq.WithReference(redis);

When I start the application locally on my mac. I get the error: Aspire.Hosting.DistributedApplicationException: Redis resource does not have endpoint annotation.

This is the same behavior when I try to reference my custom containers.

[davidfowl]

Dupe of #128

[davidfowl]

When I start the application locally on my mac. I get the error: Aspire.Hosting.DistributedApplicationException: Redis resource does not have endpoint annotation.

I'm not sure how you got this exception. Can you share all of your code and the stack trace?

[simonhoss]

This is the full code:

using Test.cloud.AppHost;
using System.Net;

var builder = DistributedApplication.CreateBuilder(args);

var redis = builder.AddRedisContainer("redis");
var rabbitMq = builder.AddRabbitMQContainer("rabbitmq");

rabbitMq.WithReference(redis);

builder.Build().Run();

Full stack trace:

      Aspire.Hosting.DistributedApplicationException: Redis resource does not have endpoint annotation.
         at Aspire.Hosting.ApplicationModel.RedisContainerResource.GetConnectionString() in /_/src/Aspire.Hosting/Redis/RedisContainerResource.cs:line 20
         at Aspire.Hosting.ResourceBuilderExtensions.<>c__DisplayClass8_0`1.<WithReference>b__0(EnvironmentCallbackContext context) in /_/src/Aspire.Hosting/Extensions/ResourceBuilderExtensions.cs:line 149
         at Aspire.Hosting.Dcp.ApplicationExecutor.CreateContainersAsync(IEnumerable`1 containerResources, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 564
         at Aspire.Hosting.Dcp.ApplicationExecutor.CreateContainerSingletonsAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 108
         at Aspire.Hosting.Dcp.ApplicationExecutor.RunApplicationAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 74
         at Aspire.Hosting.Dcp.DcpHostService.StartAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/DcpHostService.cs:line 83
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
Could not stop container 'rabbitmq': k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"containers.usvc-dev.developer.microsoft.com \"rabbitmq\" not found","reason":"NotFound","details":{"name":"rabbitmq","group":"usvc-dev.developer.microsoft.com","kind":"containers"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_DeleteClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.DeleteClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass12_0`1.<<DeleteAsync>b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 109
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 198
   at Aspire.Hosting.Dcp.ApplicationExecutor.DeleteResourcesAsync[RT](String resourceName, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 686
Could not stop service 'redis': k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services.usvc-dev.developer.microsoft.com \"redis\" not found","reason":"NotFound","details":{"name":"redis","group":"usvc-dev.developer.microsoft.com","kind":"services"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_DeleteClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.DeleteClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass12_0`1.<<DeleteAsync>b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 109
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 198
   at Aspire.Hosting.Dcp.ApplicationExecutor.DeleteResourcesAsync[RT](String resourceName, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 686
Could not stop service 'rabbitmq_tcp': k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services.usvc-dev.developer.microsoft.com \"rabbitmq_tcp\" not found","reason":"NotFound","details":{"name":"rabbitmq_tcp","group":"usvc-dev.developer.microsoft.com","kind":"services"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_DeleteClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.DeleteClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass12_0`1.<<DeleteAsync>b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 109
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 198
   at Aspire.Hosting.Dcp.ApplicationExecutor.DeleteResourcesAsync[RT](String resourceName, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 686
Could not stop service 'rabbitmq_management': k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services.usvc-dev.developer.microsoft.com \"rabbitmq_management\" not found","reason":"NotFound","details":{"name":"rabbitmq_management","group":"usvc-dev.developer.microsoft.com","kind":"services"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_DeleteClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.DeleteClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, String name, V1DeleteOptions body, Nullable`1 gracePeriodSeconds, Nullable`1 orphanDependents, String propagationPolicy, String dryRun, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass12_0`1.<<DeleteAsync>b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 109
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 198
   at Aspire.Hosting.Dcp.ApplicationExecutor.DeleteResourcesAsync[RT](String resourceName, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 686
Unhandled exception. System.AggregateException: One or more errors occurred. (Redis resource does not have endpoint annotation.)
 ---> Aspire.Hosting.DistributedApplicationException: Redis resource does not have endpoint annotation.
   at Aspire.Hosting.ApplicationModel.RedisContainerResource.GetConnectionString() in /_/src/Aspire.Hosting/Redis/RedisContainerResource.cs:line 20
   at Aspire.Hosting.ResourceBuilderExtensions.<>c__DisplayClass8_0`1.<WithReference>b__0(EnvironmentCallbackContext context) in /_/src/Aspire.Hosting/Extensions/ResourceBuilderExtensions.cs:line 149
   at Aspire.Hosting.Dcp.ApplicationExecutor.CreateContainersAsync(IEnumerable`1 containerResources, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 564
   at Aspire.Hosting.Dcp.ApplicationExecutor.CreateContainerSingletonsAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 108
   at Aspire.Hosting.Dcp.ApplicationExecutor.RunApplicationAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 74
   at Aspire.Hosting.Dcp.DcpHostService.StartAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/DcpHostService.cs:line 83
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Aspire.Hosting.DistributedApplication.RunAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/DistributedApplication.cs:line 147
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Aspire.Hosting.DistributedApplication.Run() in /_/src/Aspire.Hosting/DistributedApplication.cs:line 155
   at Program.<Main>$(String[] args) in /Users/simonhoss/Documents/entwicklung/Octopus/iot/cloud/cloud.AppHost/Program.cs:line 11

The code where the exception is thrown is here:

        if (!this.TryGetAnnotationsOfType<AllocatedEndpointAnnotation>(out var allocatedEndpoints))
        {
            throw new DistributedApplicationException("Redis resource does not have endpoint annotation.");
        }

On my system I saw that the AllocatedEndpointAnnotation are added to late. When I debug I'm able to reach this point in ApplicactionExecutor, but far to late.

                var a = new AllocatedEndpointAnnotation(
                    sp.ServiceBindingAnnotation.Name,
                    PortProtocol.ToProtocolType(svc.Spec.Protocol),
                    svc.AllocatedAddress!,
                    (int)(sp.ServiceBindingAnnotation.Port ?? svc.AllocatedPort!), // Checked by HasCompleteAddress above.
                    sp.ServiceBindingAnnotation.UriScheme
                    );

                appResource.ModelResource.Annotations.Add(a);

[davidfowl]

I see you’re using the latest builds. This was a regression introduced that we’re looking at