Make reading Kubeconfig from DCP resilient

[mitchdenny]

This PR adds some retry logic to EnsureKubernetes. As part of the process the method has become async and I'm making use of a semaphore to stop the stampede to read the file it'll only read once whilst the other threads slowly spin if they get caught up in the semaphore.

[karolz-ms]

I think a better fix would be to address this on the DCP side. Namely, produce the kubeconfig file with the following sequence:

1. Create a temporary file
2. Write the content and close the file
3. Rename the file so that it matches the name the readers expect.

This should be a pretty easy fix on DCP side

[karolz-ms]

Had a brief chat with @davidfowl , working on a DCP fix for kubeconfig access conflict.

[mitchdenny]

I think a better fix would be to address this on the DCP side. Namely, produce the kubeconfig file with the following sequence:

1. Create a temporary file
2. Write the content and close the file
3. Rename the file so that it matches the name the readers expect.

This should be a pretty easy fix on DCP side

We have to do this both on the DCP side and here. We have to add the retry logic here because we are looking for it before DCP has put the file down on disk. What you are proposing for DCP will improve things as well.

[karolz-ms]

Overall I like the direction, my main concern is super low frequency of probing for the file in this attempt.

[karolz-ms]

We have to do this both on the DCP side and here. We have to add the retry logic here because we are looking for it before DCP has put the file down on disk. What you are proposing for DCP will improve things as well.

Makes sense. I gave you some suggestions, HTH!

[mitchdenny]

@karolz-ms could you please reset your vote?

[mitchdenny]

/backport to release/8.0-preview5

[github-actions]

Started backporting to release/8.0-preview5: https://github.com/dotnet/aspire/actions/runs/8431360102

[davidfowl]

Small nit

[mitchdenny]

/backport to release/8.0-preview5

[github-actions]

Started backporting to release/8.0-preview5: https://github.com/dotnet/aspire/actions/runs/8433947069

[mitchdenny]

/backport to release/8.0-preview5

[github-actions]

Started backporting to release/8.0-preview5: https://github.com/dotnet/aspire/actions/runs/8434031136

Discussed with Karol who is out