[BUG] After a certain number of timeouts ARM ends up in a bad state where all future calls also timeout

[jfleuryStatcan]

Library name and version

Azure.ResourceManager 1.7.0

Describe the bug

After a certain number of timeouts exception resource manager ends up in a bad state where all subsequent calls will time out even if the far end is available. The application needs to be restarted.

This seems to only happen when multiple ArmClients are created. If we pass in HttpClients to each one and dispose of them when we are done the problem disappears. This seems to have something to do with all Azure SDK clients, by default, sharing a single HttpClient instance.

Expected behavior

Subsequent calls work and are not impacted by previous timeouts.

Actual behavior

All calls time out until the application is restarted.

Here is the exception we get:

Exception: ClientSecretCredential authentication failed: Request to the endpoint timed out. at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
at Azure.Identity.ClientSecretCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthorizeRequest(HttpMessage message)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachinesRestOperations.InstanceView(String subscriptionId, String resourceGroupName, String vmName, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachineResource.InstanceView(CancellationToken cancellationToken)

Reproduction Steps

int count = 20;

ArmClient client = null;

while (true)
   {
      try
         {
            Console.WriteLine(count);

            if (count > 0)
            {
               count--;

               ArmClientOptions armClientOptions = new ArmClientOptions();

               if (count > 0)
               {
                  armClientOptions.Retry.NetworkTimeout = TimeSpan.FromMilliseconds(1);
               }

               client = new ArmClient(new DefaultAzureCredential(), string.Empty, armClientOptions);
            }

            var vm = client.GetVirtualMachineResource(new ResourceIdentifier(ResourceId));

            var view = vm.InstanceView();

            Thread.Sleep(TimeSpan.FromSeconds(3));

            }
            catch (Exception ex)
            {
               Console.WriteLine(ex.ToString());
            }
         }
   }

Environment

This was run on windows 10 using .net 6 and Visual Studio 17.1.1

[christothes]

Hi @jfleuryStatcan Is there a real-world scenario behind this repro? I don't think any requests will succeed with the NetworkTimeout set to 1ms.

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jfleuryStatcan]

The first 20 times the loop runs it is supposed to timeout. After that the timeout gets set to default, so it should succeed. But it doesn't.

[jfleuryStatcan]

We've had real world events where our application has timeouts with the default setting and then it ends up in this bad state were every call timeouts until the application is restarted.

[christothes]

I think there may be a bug in your repro code. I can't repro if I eliminate the outer most if block.

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jfleuryStatcan]

Not sure why you are eliminating the outer most if block.

Here is the code written a different way. Maybe this is more clear.

        int count = 20;

        while (true)
        {
            try
            {
                Console.WriteLine(count);

                ArmClientOptions armClientOptions = new ArmClientOptions();

                if (count > 0)
                {
                    armClientOptions.Retry.NetworkTimeout = TimeSpan.FromMilliseconds(1);

                    count--;
                }

                ArmClient client = new ArmClient(new DefaultAzureCredential(), string.Empty, armClientOptions);

                var vm = client.GetVirtualMachineResource(new ResourceIdentifier(ResourceId));

                var view = vm.InstanceView();

                Thread.Sleep(TimeSpan.FromSeconds(3));

            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }

[jfleuryStatcan]

So the real world scenario here is you have an application that makes these calls every so often. Normally they work but sometimes there is a problem with the network for a period of time and you get a bunch of timeouts. Eventually the network goes back to normal but these calls keep timing out. The application ends up in a bad state and they will continually time out until the application is restarted.

So to reproduce this issue what I'm doing is setting the timeout to something ridiculously small for the first 20 times we make this call. This way we get some timeouts. After those first 20 times we sent the timeout back to the default values. It should work normally now. The fact that it failed 20 times previously should have no impact.

But this will continually timeout even when we revert to default settings. It takes the whole 5ish minutes with default settings and then times out. Because the first 20 calls timed out the system is now in a bad state and all future calls timeout.

[jfleuryStatcan]

So if we write the repro code like below then it should count down from 20 to 0 and then every 3 seconds it should output 0 and grab the instance view. We don't output exceptions that happen when the timeout was set to 1ms. When the timeout reverts to default it should work.

Instead what we get is a countdown to zero and then timeouts exceptions. All future calls timeout even with the default 00:01:40 timeout.

        int count = 20;

        while (true)
        {
            ArmClientOptions armClientOptions = new ArmClientOptions();

            try
            {
                Console.WriteLine(count);

                if (count > 0)
                {
                    armClientOptions.Retry.NetworkTimeout = TimeSpan.FromMilliseconds(1);

                    count--;
                }

                ArmClient client = new ArmClient(new DefaultAzureCredential(), string.Empty, armClientOptions);

                var vm = client.GetVirtualMachineResource(new ResourceIdentifier(ResourceId));

                var view = vm.InstanceView();

                Thread.Sleep(TimeSpan.FromSeconds(3));

            }
            catch (Exception ex)
            {
                if (armClientOptions.Retry.NetworkTimeout != TimeSpan.FromMilliseconds(1))
                {
                    Console.WriteLine(ex.ToString());
                }
            }
        }

[christothes]

With that latest code block, I see get the following output: (I added an additional console write to show that we are in the exception block regardless of timeout setting.

20
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
19
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
18
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
17
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
16
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
15
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
14
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
13
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
12
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
11
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
10
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
9
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
8
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
7
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
6
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
5
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
4
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
3
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
2
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
1
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
0
0
0
0
0
0
0
0
0

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jfleuryStatcan]

Interesting. Looks like you can't reproduce the issue. Are you using .net 6?

My output obviously looks like this:
20
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
19
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
18
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
17
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
16
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
15
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
14
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
13
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
12
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
11
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
10
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
9
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
8
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
7
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
6
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
5
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
4
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
3
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
2
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
1
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:00:00.001. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
0
Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
0

And if I run it again with the count initially set to 0 then it looks like this:
0
0
0
0
0
0

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jfleuryStatcan]

With count set to 1 or 5 I haven't been able to reproduce the issue. With count set to 10 I've been able to reproduce the issue the majority of the time but at least once or twice it didn't happen. Count of 20 seems to reproduce the issue every time for me.

I tried it again using Azure.Identity 1.10.0-beta1. I was not able to reproduce the issue. Is there a fix in that version or does it just make it harder to reproduce the issue?

[christothes]

I think it would be the newer Azure.Core that is brought in by that version of Azure.Identity. Can you try reverting that to 1.9.0 and only upgrading Azure.Core directly? https://www.nuget.org/packages/Azure.Core/1.34.0

[jfleuryStatcan]

I was able to reproduce the issue using the following packages:

<PackageReference Include="Azure.Core" Version="1.34.0" />
<PackageReference Include="Azure.Identity" Version="1.9.0" />
<PackageReference Include="Azure.ResourceManager" Version="1.7.0" />
<PackageReference Include="Azure.ResourceManager.Compute" Version="1.1.0" />

[Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700
[Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700
[Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] MSAL MSAL.NetCore with assembly version '4.49.1.0'. CorrelationId(5334c337-285c-437b-8ceb-780943a50e0d)
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False

[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.azure.com//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 5334c337-285c-437b-8ceb-780943a50e0d
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:

[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] === Token Acquisition (ClientCredentialRequest) started:
Scopes: https://management.azure.com//.default
Authority Host: login.microsoftonline.com
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] [Region discovery] Not using a regional authority.
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] [Instance Discovery] Skipping Instance discovery because it is disabled.
[Informational] Azure-Core: Request [a86ae0f3-76dc-45a1-9908-5274a7ae5a88] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
Metadata:REDACTED
x-ms-client-request-id:a86ae0f3-76dc-45a1-9908-5274a7ae5a88
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.9.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
client assembly: Azure.Identity
[Warning] Azure-Core: Error response [a86ae0f3-76dc-45a1-9908-5274a7ae5a88] 400 Bad Request (00.0s)
Server:IMDS/150.870.65.1036
Date:Wed, 19 Jul 2023 15:08:50 GMT
Content-Type:application/json; charset=utf-8
Content-Length:68

[Error] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:08:50Z - 5334c337-285c-437b-8ceb-780943a50e0d] Exception type: Azure.Identity.CredentialUnavailableException

at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Identity not found"}

Headers:
Server: IMDS/150.870.65.1036
Date: Wed, 19 Jul 2023 15:08:50 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 68

[Informational] Azure-Identity: VisualStudioCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700
[Informational] Azure-Identity: VisualStudioCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700 ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Identity: DefaultAzureCredential credential selected: Azure.Identity.VisualStudioCredential
[Informational] Azure-Identity: DefaultAzureCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d35e328c-61d9-4278-900f-c660bbc14700 ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d35e328c-61d9-4278-900f-c660bbc14700
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] attempt number 1 took 101.2s
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d35e328c-61d9-4278-900f-c660bbc14700
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] attempt number 2 took 100.0s
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d35e328c-61d9-4278-900f-c660bbc14700
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] attempt number 3 took 100.0s
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d35e328c-61d9-4278-900f-c660bbc14700
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d35e328c-61d9-4278-900f-c660bbc14700] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachinesRestOperations.InstanceView(String subscriptionId, String resourceGroupName, String vmName, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachineResource.InstanceView(CancellationToken cancellationToken)
at ConsoleApp1.Program.Main(String[] args) in C:\Users\fleujus\source\repos\ConsoleApp1\ConsoleApp1\Program.cs:line 45
---> (Inner Exception #1) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

---> (Inner Exception #2) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

---> (Inner Exception #3) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

0
[Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a
[Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a
[Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a Exception: Azure.Identity.CredentialUnavailableException (0x80131500): WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] MSAL MSAL.NetCore with assembly version '4.49.1.0'. CorrelationId(ca49b242-617a-4b52-a39b-efbf8305aa20)
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False

[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.azure.com//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - ca49b242-617a-4b52-a39b-efbf8305aa20
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:

[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] === Token Acquisition (ClientCredentialRequest) started:
Scopes: https://management.azure.com//.default
Authority Host: login.microsoftonline.com
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] [Region discovery] Not using a regional authority.
[Informational] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] [Instance Discovery] Skipping Instance discovery because it is disabled.
[Informational] Azure-Core: Request [1c97c476-e4d0-4218-a11d-e7529f1950e3] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
Metadata:REDACTED
x-ms-client-request-id:1c97c476-e4d0-4218-a11d-e7529f1950e3
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.9.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
client assembly: Azure.Identity
[Warning] Azure-Core: Error response [1c97c476-e4d0-4218-a11d-e7529f1950e3] 400 Bad Request (00.0s)
Server:IMDS/150.870.65.1036
Date:Wed, 19 Jul 2023 15:15:47 GMT
Content-Type:application/json; charset=utf-8
Content-Length:68

[Error] Azure-Identity: False MSAL 4.49.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:15:47Z - ca49b242-617a-4b52-a39b-efbf8305aa20] Exception type: Azure.Identity.CredentialUnavailableException

at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Identity not found"}

Headers:
Server: IMDS/150.870.65.1036
Date: Wed, 19 Jul 2023 15:15:47 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 68

[Informational] Azure-Identity: VisualStudioCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a
[Informational] Azure-Identity: VisualStudioCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Identity: DefaultAzureCredential credential selected: Azure.Identity.VisualStudioCredential
[Informational] Azure-Identity: DefaultAzureCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: 30142602-07f2-4ad5-875d-eba49c25045a ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Core: Request [30142602-07f2-4ad5-875d-eba49c25045a] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:30142602-07f2-4ad5-875d-eba49c25045a
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager

[jfleuryStatcan]

I was able to reproduce the issue with the beta version of Identity as well.

<PackageReference Include="Azure.Core" Version="1.34.0" />
<PackageReference Include="Azure.Identity" Version="1.10.0-beta.1" />
<PackageReference Include="Azure.ResourceManager" Version="1.7.0" />
<PackageReference Include="Azure.ResourceManager.Compute" Version="1.1.0" />

[Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8
[Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8
[Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8
[Informational] Azure-Identity: WorkloadIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] MSAL MSAL.NetCore with assembly version '4.54.1.0'. CorrelationId(21b1ec5c-7a3f-4c01-97dc-26588147f083)
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False

[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.azure.com//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 21b1ec5c-7a3f-4c01-97dc-26588147f083
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:

[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] === Token Acquisition (ClientCredentialRequest) started:
Scopes: https://management.azure.com//.default
Authority Host: login.microsoftonline.com
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] [Region discovery] Not using a regional authority.
[Informational] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] [Instance Discovery] Skipping Instance discovery because it is disabled.
[Informational] Azure-Core: Request [f8301038-6108-4281-a93b-353393d03da4] GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
Metadata:REDACTED
x-ms-client-request-id:f8301038-6108-4281-a93b-353393d03da4
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.10.0-beta.1 (.NET 6.0.18; Microsoft Windows 10.0.19044)
client assembly: Azure.Identity
[Warning] Azure-Core: Error response [f8301038-6108-4281-a93b-353393d03da4] 400 Bad Request (00.0s)
Server:IMDS/150.870.65.1036
Date:Wed, 19 Jul 2023 15:32:06 GMT
Content-Type:application/json; charset=utf-8
Content-Length:68

[Error] Azure-Identity: False MSAL 4.54.1.0 MSAL.NetCore .NET 6.0.18 Microsoft Windows 10.0.19044 [2023-07-19 15:32:07Z - 21b1ec5c-7a3f-4c01-97dc-26588147f083] Exception type: Azure.Identity.CredentialUnavailableException

at Azure.Identity.ImdsManagedIdentitySource.HandleResponseAsync(Boolean async, TokenRequestContext context, Response response, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ManagedIdentityClient.AppTokenProviderImpl(AppTokenProviderParameters parameters)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.SendTokenRequestToProviderAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8 Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Identity not found"}

Headers:
Server: IMDS/150.870.65.1036
Date: Wed, 19 Jul 2023 15:32:06 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 68

[Informational] Azure-Identity: VisualStudioCredential.GetToken invoked. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8
[Informational] Azure-Identity: VisualStudioCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8 ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Identity: DefaultAzureCredential credential selected: Azure.Identity.VisualStudioCredential
[Informational] Azure-Identity: DefaultAzureCredential.GetToken succeeded. Scopes: [ https://management.azure.com//.default ] ParentRequestId: d5cde929-b9d7-4569-9357-8298ecfb6ca8 ExpiresOn: 2023-07-19T16:31:57.0000000+00:00
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d5cde929-b9d7-4569-9357-8298ecfb6ca8
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] attempt number 1 took 101.2s
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d5cde929-b9d7-4569-9357-8298ecfb6ca8
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] attempt number 2 took 100.0s
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d5cde929-b9d7-4569-9357-8298ecfb6ca8
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] attempt number 3 took 100.0s
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] GET https://management.azure.com/subscriptions/4e580731-fa3e-4957-954a-34d4b1da003e/resourceGroups/hosting-sas-census-rhel8-v94-production-rg/providers/Microsoft.Compute/virtualMachines/stpcsas-cts-807/instanceView?api-version=2022-11-01
Accept:application/json
x-ms-client-request-id:d5cde929-b9d7-4569-9357-8298ecfb6ca8
x-ms-return-client-request-id:true
User-Agent:azsdk-net-ResourceManager.Compute/1.1.0 (.NET 6.0.18; Microsoft Windows 10.0.19044)
Authorization:REDACTED
client assembly: Azure.ResourceManager
[Informational] Azure-Core: Request [d5cde929-b9d7-4569-9357-8298ecfb6ca8] exception System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachinesRestOperations.InstanceView(String subscriptionId, String resourceGroupName, String vmName, CancellationToken cancellationToken)
at Azure.ResourceManager.Compute.VirtualMachineResource.InstanceView(CancellationToken cancellationToken)
at ConsoleApp1.Program.Main(String[] args) in C:\Users\fleujus\source\repos\ConsoleApp1\ConsoleApp1\Program.cs:line 49
---> (Inner Exception #1) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

---> (Inner Exception #2) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

---> (Inner Exception #3) System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---

[christothes]

Thanks - just to clarify, is there a config that you still cannot reproduce with?

[jfleuryStatcan]

I have yet to reproduce the problem with .net core 3.1 or .net 5. It's hard to say if it's just easier to reproduce with .net 6 or if the problem is really with .net 6. I'm going to try .net 7 and .net 8 shortly.

Are you able reproduce the problem at all with the different package versions?

[christothes]

Sorry, I meant can you still reproduce with all the package version permutations we tried? I thought there was one combo that wouldn't reproduce for you even on net 6.0

[jfleuryStatcan]

I was just able to reproduce the issue the following settings as well. I think that's just about every permutation now. Using newer versions of Core/Identity does seems to make it harder to reproduce but not impossible.

	<PackageReference Include="Azure.Identity" Version="1.10.0-beta.1" />
	<PackageReference Include="Azure.ResourceManager" Version="1.7.0" />
	<PackageReference Include="Azure.ResourceManager.Compute" Version="1.1.0" />

[christothes]

I'm afraid that I'm unable to repro, even with the older Azure.Identity. Just out of curiosity, in your actual application, are you using singletons for the ArmClient and credential?

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jfleuryStatcan]

We are not using singletons. When we first started investigating this we noticed that it's recommended to use singletons as a best practice. So that's something we've been planning on trying. We should actually be deploying that fix tomorrow morning if everything goes well.

Another idea I had around this was that the ARMClients are using a singleton HttpClient instance. So if one failed call was affecting the next I thought maybe using a new HttpClient instance each time and disposing of it may help. We tried that yesterday and it did not work. Hopefully using a singleton ARMClient fixes the problem.

[christothes]

Unless you are configuring a custom transport via ArmClientOptions, the singleton probably won't make a big difference with regard to the Http client behavior. Under the covers, we use a singleton HttpClient for all clients in the same process, unless a custom transport is provided. It will consume fewer resources, however.

The single instance of HttpClient is exactly what should be happening to efficiently use the connection pool. See this blog post for details.

[github-actions]

Hi @jfleuryStatcan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @jfleuryStatcan, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[jfleuryStatcan]

It seems like using a singleton for ARMClient, ArmClientOptions and HttpClient solved our problems. At least we haven't had any exceptions or lockups in the last three weeks.