Endpoint allocation can get stuck some times

[radical]

This code in ApplicationExecutor.CreateServicesAsync should have a timeout to either fail or at least log a message that it is still waiting on address allocation for some services, and list those services.

aspire/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs

Lines 826 to 845 in 93a1efb

	// We do not specify the initial list version, so the watcher will give us all updates to Service objects.
	IAsyncEnumerable<(WatchEventType, Service)> serviceChangeEnumerator = kubernetesService.WatchAsync<Service>(cancellationToken: cancellationToken);
	await foreach (var (evt, updated) in serviceChangeEnumerator)
	{
	if (evt == WatchEventType.Bookmark) { continue; } // Bookmarks do not contain any data.
	var srvResource = needAddressAllocated.Where(sr => sr.Service.Metadata.Name == updated.Metadata.Name).FirstOrDefault();
	if (srvResource == null) { continue; } // This service most likely already has full address information, so it is not on needAddressAllocated list.
	if (updated.HasCompleteAddress || updated.Spec.AddressAllocationMode == AddressAllocationModes.Proxyless)
	{
	srvResource.Service.ApplyAddressInfoFrom(updated);
	needAddressAllocated.Remove(srvResource);
	}
	if (needAddressAllocated.Count == 0)
	{
	return; // We are done
	}
	}

This was hit in running Aspire.EndToEnd.Tests where 1/20 runs fails where it gets stuck waiting for a service to get allocated.

In the above run it is stuck waiting for `redis`, till the test infrastructure times out after 5 minutes.

Full log. And corresponding dcp logs are here in logs/dcp-diag-logs, and the corresponding build.

Note that this is not specific to redis.

cc @karolz-ms @eerhardt

[karolz-ms]

CC @dbreshears @danegsta

In a way similar to what @mitchdenny has done, we should probably retry this watch a few times before timing out. Alternative (or last-ditch effort, we could do both) could be to explicitly query the remaining resources and see if they have the address information we are looking for.