build(publish): parallel code gen on azure-sdk-for-net

[archerzz]

• split job Update_azure_sdk_for_net into 3 stages:
  • Update_Generator_Versions: change the generator versions only, create the first commit to new branch auto-update-autorest-${Version}
  • Update_Codes_X_Y and Update_Samples_X_Y: update SDK codes or sample codes from service of which name starts with X to Y.
  • Create_PR: create PR from branch auto-update-autorest-${Version} to main
• add retry for Update_Codes_X_Y and Update_Samples_X_Y

resolve #3067 #3068

• trial run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2201624&view=results
• PR generated: Update AutoRest C# version to 3.0.0-beta.20230218.6 azure-sdk-for-net#34361

---

Checklist

To ensure a quick review and merge, please ensure:

• The PR has a understandable title and description explaining the why and what.
• The PR is opened in draft if not ready for review yet.
  • If opened in draft, please allocate sufficient time (24 hours) after moving out of draft for review
• The branch is recent enough to not have merge conflicts upon creation.

Ready to Land?

• Build is completely green
  • Submissions with test failures require tracking issue and approval of a CODEOWNER
• At least one +1 review by a CODEOWNER
• All -1 reviews are confirmed resolved by the reviewer
  • Override/Marking reviews stale must be discussed with CODEOWNERS first

[archerzz]

Other benefits:

• Re-running the failed tasks will be much easier. We don't need to start over from the very beginning again.
• For PR with thousands of changes, people can choose to review each commit (there are 12 commits in total).
• Each change on auotrest.csharp is now separately published to azure-sdk-for-net. So if a change screws the azure-sdk-for-net, people can still merge all previous changes.

[m-nash]

@mikeharder can you please take a look as well since you are helping in the same area. This gets us from 3+ hours to roughly 35 minutes on the regen portion of the publish pipeline.

If you have even more improvements lets open those as trackers so we can make incremental progress but if you see any major issues with this let us know.

[mikeharder]

@m-nash, @archerzz: Are there example builds before and after this change I can look at?

I see an example build at the top of this PR: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2201624&view=results. I assume I should just compare this to a similar build from another PR?

[archerzz]

@m-nash, @archerzz: Are there example builds before and after this change I can look at?

I see an example build at the top of this PR: dev.azure.com/azure-sdk/internal/_build/results?buildId=2201624&view=results. I assume I should just compare this to a similar build from another PR?

@mikeharder Recently we don't change the pipeline except this PR. So any build on feature/v3 branch should be before this change, like this one.

[archerzz]

Here is the comparison of existing pipeline and new pipeline.

Existing Pipeline

• latest run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2210555&view=results
• PR created (549 files changed): Update AutoRest C# version azure-sdk-for-net#34465

New Pipeline

• latest run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2211526&view=results
• PR created (549 files changed): Update AutoRest C# version to 3.0.0-beta.20230222.4 azure-sdk-for-net#34495

The changes in two PR are actually identical.