Transition Windows and Ubuntu agent pools to consume 2022-Compliant images

[scbedd]

Our repositories run CI from agent pools configured with Windows-20XXTLS and Ubuntu-20XXTLS images. To address shifts in the security landscape, we will need to mandate our build agents run the the compliant 1ES agents.

In 1ES agent selection, these all have that keyword Compliant in the image name.

• This transition will need to happen for both windows and linux agents.
• Included Repos
  • azure-sdk-for-java
  • azure-sdk-for-net
  • azure-sdk-for-js
  • azure-sdk-for-python
  • azure-sdk-for-go
  • azure-sdk-for-cpp
  • azure-sdk-for-c
  • azure-sdk-for-ios
  • azure-sdk-for-android

Due to the greater restrictions in the image, this should NOT be assumed to be an easy transition

[sima-zhu]

Created the new pool for win-2022 complant.

[sima-zhu]

1. Create a MMS-Win-2022 image.
2. Create a pool using the image.

[konrad-jamrozik]

@weshaggard we talked about me updating our images to Ubuntu 22.04 in all our language repos. I captured it in this work item:

• Update Ubuntu images used in all our language and other SDK repos to 22.04, where applicable #5058

But seems to me that we actually need to do this work instead, assigned to @sima-zhu ?

I.e. it is not only a version bump, but also update to compliant images, and as @scbedd mentioned:

Due to the greater restrictions in the image, this should NOT be assumed to be an easy transition

Or are we dealing here with two separate non-overlapping sets of pools, and both Sima and I can do our work independently?

[weshaggard]

Or are we dealing here with two separate non-overlapping sets of pools, and both Sima and I can do our work independently?

They are separate pools one is the Windows 22 and the other Ubuntu 22. They should be able to be done independently.

[konrad-jamrozik]

Picking this up now. The migration to compliant windows images needs to be completed before 3/1/2023, per internal email thread with @mikeharder. That email thread contains details on what work was already done, and what remains.

[konrad-jamrozik]

There have been changes in image names and what we will exactly do.

Some context is available only in my private conversations with Mike Harder, but there is also MS-internal context provided here:

Mike Harder: 1ES Hosted Pool image name changes
posted in Azure SDK / Engineering System 🛠️ at Wednesday, February 15, 2023 12:10 PM

Mike Harder: Migrate all pipelines off Ubuntu 18.04 by 4/1/2023
posted in Azure SDK / Engineering System 🛠️ at Wednesday, February 15, 2023 12:32 PM

[konrad-jamrozik]

Status update 2/18/2023

Updating Windows images to compliant (deadline: 3/1/2023)

For now the focus was on updating the agent pools azsdk-pool-mms-win-2019-general and azsdk-pool-mms-win-2019-storage to their compliant 2022 equivalents: azsdk-pool-mms-win-2022-general and azsdk-pool-mms-win-2022-storage. This work needs to be done by March 1st, 2023, at which point the pools will be force-updated by 1ES to use compliant images. Update of Ubuntu images has to be done by April 1st, per #5472.

This work is almost complete; there is one repository outstanding that needs to be updated, azure-rest-api-specs-pipeline. Details on that here:

• https://github.com/Azure/azure-rest-api-specs-pipeline/pull/75#issuecomment-1435801145

There are few cases where we couldn't update azsdk-pool-mms-win-2019-general to 2022. They can be found by using this query for new GitHub search (org:Azure azsdk-pool-mms-win-2019-general repo:Azure/azure-sdk-for-c OR repo:Azure/azure-sdk-for-cpp OR repo:Azure/azure-sdk-for-c-pr). Specifically, the occurrences are in these repos:

• Azure/azure-sdk-for-c
• Azure/azure-sdk-for-c-pr
• Azure/azure-sdk-for-cpp

There are few repositories that reference the pool azsdk-pool-mms-win-2019-general but don't actually schedule any builds using it. These are:

• Azure/azure-sdk-tool
• Azure/embedded-wireless-framework

Updating vmImage to windows-2022

Per Mike's post mentioned in the comment above, the secondary goal of current work was to update pool vmImage (and related, like OSVmImage; see also issue #5494) values to windows-2019 or windows-2022, as appropriate.

Note that these vmImage strings are ignored by the pools and hence do not influence if the VMs we are using are compliant or not. Instead, given pool has its own declaration of the image it uses.

For updating the vmImage strings, in practice this meant that while updating the pool to 2022 as described above, I also did search and replace for:

• MMS2021Compliant
• MMS2021
• MMS2022Compliant
• MMS2022
  -->
• windows-2022

and for cases that couldn't be updated to 2022:

• MMS2021Compliant
• MMS2021
  -->
• windows-2019

There is currently one open PR pending approval for such image update:

• Where applicable, update Windows pools used to azsdk-pool-mms-win-2022-general and rename vmImage to the windows-20xx format autorest.csharp#3152

However there are still occurrences of these strings in repos in Azure org as I wasn't yet exhaustive in replacing these string everywhere - I prioritized updating the pools to be compliant before March 1st. Here is a query in the new GitHub search (org:Azure (path:*.y*ml OR path:*.json) MMS2022 OR MMS2021)) to find all occurrences of these old strings.

Next steps

For updating to compliant images

At this point we should update the image used by azsdk-pool-mms-win-2019-general to complaint and observe if builds scheduled from these repos still work. The builds can be reviewed on the pool page.

@mikeharder I do not know how to do update the 2019-general pool image to compliant, I probably don't have permissions, and you hinted this might be a nontrivial process. Would you mind pairing up with me to get the 2019-general pool image updated to complaint one?

For updating the vmImage string

Per the query mentioned above, we should find and replace all the remaining occurrences. I will take care of this.

Outstanding questions

@mikeharder I assume that strings like these should be left alone:

https://github.com/Azure/azure-functions-durable-extension/blob/2ec836c9555e10845b7e244d7ce7f19bb0f7b460/azure-pipelines-release-dotnet-isolated.yml#L8

?

I.e. I should not change it to windows-2022 as it is not using our pools.

[konrad-jamrozik]

Per my chat with @mikeharder:

• We should not update MMS2022TLS on pools that aren't ours, albeit it is a bit curious that he repo is using MMS2022TLS; likely they copy-pasted from our OS repos.
• Pool update to a compliant image is straightforward, one just needs to update relevant Azure portal 1ES hosted pool at https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/<subId>/resourceGroups/<resGroupId>/providers/<providerId>/hostedpools/azsdk-pool-mms-win-2019-general/pool.

[konrad-jamrozik]

Status update 2/21/2023

Since the last update we have updated the azsdk-pool-mms-win-2019-general pool to use compliant image.

I have also made and merged 2 more PRs against repos in Azure DevOps orgs:

• https://dev.azure.com/azure-sdk/internal/_git/azure-sdk-build-tools/pullrequest/718
• https://dev.azure.com/azure-sdk/internal/_git/azure-sdk-devops/pullrequest/719

These instances needing updating have been found by doing org-wide search for win-2019:
https://dev.azure.com/azure-sdk/internal/_search?action=contents&type=code&text=win-2019&filters=ProjectFilters%7Binternal

I have also reached out over email to owners of following build definitions:

• test-repo-billy.azure-rest-api-specs-pipeline, as this repo appears to be private and inaccessible
• python-pr - experimentation, as it is having builds made from user branch that needs to be rebased on main.
  • Update 2/23/2023 4:51 PM PST: this has been fixed. The branch has been rebased.

I am currently waiting on reply from both. Once these last 2 issues are fixed, the work should be complete. @mikeharder has also made announcement on our Teams channel:

Mike Harder: Windows 2019 agents upgraded to "Compliant" image
posted in Azure SDK / Engineering System 🛠️ at Tuesday, February 21, 2023 12:52 PM

[konrad-jamrozik]

Status update 3/2/2023

Since the last update there is only one build definition that uses windows image that needs updating left, originating from this repository:

• test-repo-billy.azure-rest-api-specs-pipeline

Still waiting for an email reply to get this fixed.

Besides, all other Windows images have been updated to compliant (whether 2019 or 2022) and no issues have been reported so far.

[konrad-jamrozik]

Status update 3/3/2022

Windows update completed

The update of Windows images to compliant, and to 2022 where possible, is now complete.

This is how I addressed the remaining test-repo-billy.azure-rest-api-specs-pipeline update:

@weshaggard gave me access to the org of test-repo-billy.azure-rest-api-specs-pipeline.

Searching for pool in the repo shows the only occurrences of Windows pools use the 2022 image. Also there appear to be no recent 2019 pool usage by this pipeline.

Looking at the pipeline runs, like this one, I suspect the obsolete pool usages were coming from CredScan and PoliCheck and have been mirrored to this repo once this change was made:

• https://github.com/Azure/azure-rest-api-specs-pipeline/pull/75#issuecomment-1439014108

Next: Ubuntu update

Next work to do here is:

• Migrate all pipelines off Ubuntu 18.04 by 4/1/2023 #5472

[konrad-jamrozik]

With the completion of this work item just now:

• Migrate all pipelines off Ubuntu 18.04 by 4/1/2023 #5472

The current issue is also resolved. Closing.