Add pool definition to prepare-pipelines.yml set to Ubuntu 20.04, and wrap the steps in a job #4911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
konrad-jamrozik
requested review from
sima-zhu,
weshaggard and
benbp
as code owners
|
The following pipelines have been queued for testing: |
konrad-jamrozik
added
the
Central-EngSys
weshaggard
reviewed
Dec 9, 2022
weshaggard
reviewed
Dec 9, 2022
konrad-jamrozik
force-pushed
the
users/kojamroz/iss_4880_fix_pool
branch
from
ea8d29c
to
5beac6f
Compare
|
The following pipelines have been queued for testing: |
ghost
pushed a commit
that referenced
this pull request
Dec 12, 2022
… to use Ubuntu 20.04 (#4930) This PR, in tandem with PRs #4915 and #4916, addresses #4888. This PR subsumes the abandoned PR of #4911. Specifically, this PR ensures that `prepare-pipelines.yml` is configured to use specific `pool`, set to use Ubuntu 20.04: ``` pool: name: azsdk-pool-mms-ubuntu-2004-general vmImage: MMSUbuntu20.04 ``` Without the [pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) definition pointing to Ubuntu 20.04, the steps within the file used an implicit default, which resolves to `ubuntu-latest`. This caused breakage as explained in #4888. Setting the pool to Ubuntu 20.04 will allow us to: - Immediately unblock the language repos `prepare-pipelines.yml` pipelines [1], as Ubuntu 20.04 has the .NET Core version used by it [2]; - Allow us to migrate our tooling to .NET 6 as both Ubuntu 20.04 and `ubuntu-latest`, which is Ubuntu 22.04 [3], have .NET on them [4]. This step is accomplished by PRs #4916 and #4915; - Allow us to migrate to Ubuntu 22.04 once we migrate to .NET 6. ## Technical considerations of the changes - The current `prepare-pipelines.yml` has only [steps](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps?view=azure-pipelines) definition, which does not support `pool` definition. One needs to wrap the steps in [job](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job?view=azure-pipelines) definition to support `pool` [5]. - Because the `.yml` file is located in `eng/common/pipelines/templates/steps/` directory (note the `steps`), we cannot wrap the steps in job and leave it there; the introduction of `job` necessitates introducing the changes in `eng/common/pipelines/templates/jobs/` directory instead (note the `jobs`), which is done in this PR. - Because this PR introduces a new file, once this PR is merged, the language-specific `prepare-pipelines.yml` language pipelines need to be rewired to point to the newly introduced file. This approach was proposed in #4911 (comment). - Language repos PRs doing the rewiring: Azure/azure-sdk-for-net#32999 ... more to come! ## Testing done I have confirmed the proposed modification to the `.yml` file will work [by modifying the existing `prepare-pipelines.yml` directly](Azure/azure-sdk-for-java@f8cef4c) to have exactly the same contents as the `/jobs/prepare-pipelines.yml` introduced in this PR, and observing the [relevant build succeeds on azure-sdk-for-java](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2044120&view=logs&j=7f42699e-eeb0-56c8-40c2-c88ae4093e4f&t=a02502cb-238e-5603-14ee-8bb7bd07f0c6):  while it failed before this change:  # Footnotes [1] for example, [internal / java / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2158&_a=summary) pipeline, or [internal / net / prepare-pipelines](https://dev.azure.com/azure-sdk/internal/_build?definitionId=2179) pipeline. [2] as evidenced by the [Ubuntu 20.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#net-core-sdk), it has .NET Core 3.1. The pool definition value points to a 1ES hosted image which has the software listed on that page, as explained by this comment: #4888 (comment). [3] See this comment: #4888 (comment). [4] as evidenced by the [Ubuntu 22.04 image software page](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#net-core-sdk). [5] Observe that on [pool definition](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/pool?view=azure-pipelines) YAML reference page, we can read: > Properties that use this definition: pipeline.pool, stages.stage.pool, jobs.job.pool, jobs.deployment.pool. i.e. `steps.pool` is not listed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
UPDATE 12/9/2022 2:46 PM PST: this PR has been abandoned. It is subsumed by PR #4930.
This PR, in tandem with PR #4915, addresses #4888 by doing the following:
jobis namedPreparePipelines, mimicking the nameGeneratePipelinesas seen in pipeline-generation.yml.pooldefinition with value copy-pasted from pipeline-generation.yml, which points to the ubuntu 20.04 1ES hosted pool:By setting the image to
MMSUbuntu20.04, which hasnet6.0installed, and ensuring the tool used by the pipeline,PipelineGenerator, is usingnet6.0, we ensure the pipeline will work as expected.For more detailed analysis on why this combination of
PipelineGeneratorSDK version and pool images, see #4915 (comment).Testing
I am not going to merge this PR until I test if it is going to work in expected way. I will do so by submitting PRs against some target
azure-sdk-for-*repos, changing their derivedprepare-pipelines.yml. I will provide an update here once done with these tests. Note that the file updates made by these test PRs will be transient, as they will be overridden by changes made by this PR once fully deployed, as explained by the Common Engineering System doc.UPDATE 12/8/2022 9:24 PM PST: indeed, wrapping the steps into a job and setting the pool makes the pipeline pass, as evidenced by this build on azure-sdk-for-java, from my branch.