Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job matrix generation scripts #16450

Merged
merged 1 commit into from
Feb 1, 2021
Merged

Conversation

benbp
Copy link
Member

@benbp benbp commented Feb 1, 2021

This adds scripts, docs and samples supporting dynamic, cross-product matrix generation for azure pipeline jobs.
It aims to replicate the cross-product matrix functionality in github actions,
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, parameter grouping and matrix filters.

These changes are live for the .NET tests and coming soon for Java, so this PR is just a port of the code that already exists in the those repositories. The intent is to standardize it across the language repos, and then migrate all the common code into azure-sdk-tools. I will make a follow-on PR with the actual configuration changes to use this functionality in the pipelines, because I want to keep the changes separate to make reviewing/reverting easier.

This functionality is made possible by the ability for the azure pipelines yaml to take a dynamic variable as an input
for a job matrix definition
(see the code sample at the bottom of the linked section).

See the README.md file for more details on the config file syntax and usage, as well as implementation details.

The tests (test-matrix-functions.tests.ps1) contain a lot of detail on expected data structures at various processing stages. The -test-matrix-functions.ps1` file could perhaps be split up or use some more organization, so let me know if it's hard to navigate.

Example (from a Java scenario...you get the idea):

{
  "displayNames": {
    "true": "TestFromSource"
  },
  "matrix": {
    "Agent": {
      "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
      "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
      "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
    },
    "JavaTestVersion": [ "1.8", "1.11" ],
    "AZURE_TEST_HTTP_CLIENTS": [ "okhttp", "netty" ]
  },
  "include": [
    {
      "Agent": {
          "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
      },
      "JavaTestVersion": "1.11",
      "AZURE_TEST_HTTP_CLIENTS": "netty",
      "TestFromSource": true
    }
  ]
}

Sparse matrix job generation in a pipeline: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=705622&view=results

image

Related discussion: microsoft/azure-pipelines-yaml#20

@benbp benbp self-assigned this Feb 1, 2021
@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Feb 1, 2021
@benbp benbp merged commit 7f9136a into Azure:master Feb 1, 2021
@benbp benbp deleted the benbp/job-matrix branch February 1, 2021 19:08
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Feb 2, 2021
…into analyze_redesign

* 'master' of https://github.com/Azure/azure-sdk-for-python: (35 commits)
  Sync eng/common directory with azure-sdk-tools for PR 1353 (Azure#16465)
  Normalize the package name for Doc.Ms readme (Azure#16401)
  fix changelog and version (Azure#16445)
  we should always run integration to publish from our artifacts. if the build pipeline crashed too early in the pipeline, this will fail regardless, due to inability to pull the artifacts down (Azure#15058)
  Fix DateTime bug (Azure#16456)
  Resolve Regression Failures (Azure#16455)
  [text analytics] Expose 'string_index_type' parameter in all service client methods where applicable (Azure#16412)
  adding devtools to the appconfig dev_reqs to solve python core issue (Azure#16381)
  Copy job matrix functionality (Azure#16450)
  Add APIView KV variable group to prepare pipelines bot (Azure#16451)
  [Datalake] Added support for PurePosixPath (Azure#16400)
  Regenerate baseline because last one break. (Azure#16415)
  adding step to test for crlf line endings (Azure#16398)
  [Datalake] Removed list_paths manual paging and deserialization (Azure#16309)
  Sync eng/common directory with azure-sdk-tools for PR 1351 (Azure#16448)
  Update auto_codegen.py (Azure#16443)
  First release purview (Azure#16440)
  1ES pools update for release pipeline. (Azure#16419)
  Add Cloud environment for Teams user (Azure#16359)
  Sync eng/common directory with azure-sdk-tools for PR 1345 (Azure#16404)
  ...
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Feb 2, 2021
…into enum-meta

* 'master' of https://github.com/Azure/azure-sdk-for-python: (128 commits)
  Communication identity api redesign (Azure#16420)
  fix EH samples and docs (Azure#16457)
  T2 redisenterprise 2021 02 02 (Azure#16472)
  Update automation_generate.sh (Azure#16470)
  Sync eng/common directory with azure-sdk-tools for PR 1353 (Azure#16465)
  Normalize the package name for Doc.Ms readme (Azure#16401)
  fix changelog and version (Azure#16445)
  we should always run integration to publish from our artifacts. if the build pipeline crashed too early in the pipeline, this will fail regardless, due to inability to pull the artifacts down (Azure#15058)
  Fix DateTime bug (Azure#16456)
  Resolve Regression Failures (Azure#16455)
  [text analytics] Expose 'string_index_type' parameter in all service client methods where applicable (Azure#16412)
  adding devtools to the appconfig dev_reqs to solve python core issue (Azure#16381)
  Copy job matrix functionality (Azure#16450)
  Add APIView KV variable group to prepare pipelines bot (Azure#16451)
  [Datalake] Added support for PurePosixPath (Azure#16400)
  Regenerate baseline because last one break. (Azure#16415)
  adding step to test for crlf line endings (Azure#16398)
  [Datalake] Removed list_paths manual paging and deserialization (Azure#16309)
  Sync eng/common directory with azure-sdk-tools for PR 1351 (Azure#16448)
  Update auto_codegen.py (Azure#16443)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants