Reduce dev build times

[rajeshkamal5050]

• Triage time spent "waiting for machines to be available" @danieljurek
• Fix Test_CLI_Aspire_Deploy with recordings @weikanglim fix CI perf regression with Test_CLI_Aspire_Deploy  #4323

[weikanglim]

Example pool waiting message:

This agent cloud does not support more specific status information./nPool provider seems slow, we notified pool provider at 9/18/2024 11:22:50 PM, still agent has not started processing it. Pool provider delay(in mins): 5.49405499333333

Happened on ValidateCrossCompileWindowsARM64

[danieljurek]

Windows machines take a "long time" to recycle in our Azure DevOps pools. So if all machines in the pool are in use or recycling after use there can be a significant delay.

The fix in this case is to increase the size of the pool or change validation to run less frequently.

[weikanglim]

Just wondering, is there a dashboard tracking "average added queue time (per build)"? I feel like this could be a good SLA to define -- optimally, it should be under 5 minutes.

[rajeshkamal5050]

@danieljurek moving this to Backlog. Please feel free to pull it in when you are working on it.