Migrate to 1ES templates for internal builds

[MiYanni]

Summary

This effort is to migrate to the 1ES pipeline templates (1ES PT). This repo shows a migration to these templates with a repo that uses Arcade. Arcade pushed an update for their pipeline templates that work with the 1ES templates. This PR extends the 1ES PT and updates the internal pipeline to use the updated Arcade templates. Our internal pipeline uses .vsts-ci.yml and our external (PR) pipeline uses .vsts-pr.yml.

Below, I've included my notes on the process as I went through the process of adopting these templates. Note that these notes may have some specifics that apply to only Arcade repos or only this repo, so these instructions aren't complete for every situation.

High-Level Notes

• I used this PR as a reference: https://dnceng.visualstudio.com/internal/_git/dotnet-symreader/pullrequest/37739
  • This isn't a great reference as some of the information is wrong/inaccurate, so take it with a grain of salt
  • This is a bad example. Use this PR itself as an example.
• In VSCode/your editor of choice, turn on View -> Appearance -> Render Whitespace
  • This helps with seeing correct indentation, which is required for YAML.
• Verify you have the ability to create internal branches on your repo for testing
• Prior to this PR being merged, switch the PR pipeline to new pr yml file
• You may need to do split-diff view AND ignore whitespace changes on this PR to view it, since indentation gets shifted around

Process

Extend 1ES PT

• Add the 1ESPipelineTemplates repo as a resource
  • Note: You may already have a resources: node or even repositories: under it. This is an additional repository object in that array.

Example

resources:
  repositories:
  - repository: 1esPipelines
    type: git
    name: 1ESPipelineTemplates/1ESPipelineTemplates
    ref: refs/tags/release

• Extend the 1ES template and indent stages: as a parameters: property
  • The template: is referencing the repository above and the YAML file in that repo
  • The sourceAnalysisPool: sets the pool information for running the SDL tools. See the section on updating build pools for more context.
  • The customBuildTags: adds tags to your pipeline runs for 1ES.

Before

stages:
- stage: Build
  jobs:
...

After

extends:
  template: v1/1ES.Official.PipelineTemplate.yml@1esPipelines
  parameters:
    sdl:
      sourceAnalysisPool:
        name: $(DncEngInternalBuildPool)
        image: 1es-windows-2022
        os: windows
    customBuildTags:
    - ES365AIMigrationTooling
    stages:
    - stage: Build
      jobs:
...

Update local template paths

• Replace ALL Arcade template references of path /eng/common/templates with /eng/common/templates-official
• Add @self to the end of all local (in-repo) template paths (including the Arcade templates)
  • The @self lets the 1ES PT template know to access this file path from within your repository, not the 1ES PT repo.
  • More info on that here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops&pivots=templates-includes#use-other-repositories

Arcade Templates

Before

template: /eng/common/templates/job/job.yml

After

template: /eng/common/templates-official/job/job.yml@self

Non-Arcade Templates

Before

template: eng/build.yml

After

template: eng/build.yml@self

Update pool information

• Update the image information
  • For Windows, images starting with windows.vs2022 or 1es will work with 1ES PT.
  • For Linux, images starting with 1es will work with 1ES PT.
  • For MacOS, the only working image is from the public Azure Pipelines pool. Additional information (see example below) is required for using this with 1ES PT.
  • Image information can be found here: https://helix.dot.net/#1esPools

Before (Windows)

name: $(DncEngInternalBuildPool)
demands: ImageOverride -equals windows.vs2022preview.amd64

After (Windows)

name: $(DncEngInternalBuildPool)
image: windows.vs2022preview.amd64
os: windows

Before (Linux)

name: $(DncEngInternalBuildPool)
demands: ImageOverride -equals build.ubuntu.1804.amd64

After (Linux)

name: $(DncEngInternalBuildPool)
image: 1es-ubuntu-1804
os: linux

Before (Mac)

vmImage: 'macOS-latest'

After (Mac)

name: Azure Pipelines
image: macOS-latest
os: macOS

• If your pipeline did not use the $(DncEngInternalBuildPool) variable for the pool name previously, you will need to add this variable template to your variables: section

Example

variables:
...
- template: /eng/common/templates-official/variables/pool-providers.yml

Update publish tasks

• Replace publish: tasks with task: 1ES.PublishPipelineArtifact@1

Before

- publish: $(Build.SourcesDirectory)\eng\buildConfiguration
  artifact: buildConfiguration
  displayName: Publish Build Config

After

- task: 1ES.PublishPipelineArtifact@1
  displayName: Publish Build Config
  inputs:
    targetPath: $(Build.SourcesDirectory)\eng\buildConfiguration
    artifactName: buildConfiguration

• Replace the task PublishBuildArtifacts@1 with 1ES.PublishBuildArtifacts@1 (just prepend 1ES. to the name)
  • No task values should need to change beyond the task name
  • If you have task errors, you may need to investigate them accordingly

Other Notes (but might apply to you)

• I needed to duplicate the local eng/build.yml to eng/build-pr.yml so that the PR pipeline uses the original (non-1ES PT) logic
  • I updated the .vsts-pr.yml to reference the eng/build-pr.yml template.
  • Different repos may need to handle their local templates differently, depending on how they are used and what they contain.

Builds

• https://dnceng.visualstudio.com/internal/_build/results?buildId=2403255
• https://dnceng.visualstudio.com/internal/_build/results?buildId=2404420
• https://dnceng.visualstudio.com/internal/_build/results?buildId=2405780

[MiYanni]

I've added these pipeline link comments to the top of the YAML files so it is easy to get to them.

[MiYanni]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[Forgind]

@self?

[MiYanni]

@self is only needed when you're providing a path as a parameter for a file in another repo, such as the 1ES template. Here, this file is in this repo and the path is in the repo, so it doesn't need it.

Meaning, template: v1/1ES.Official.PipelineTemplate.yml@1esPipelines sets the context to @1esPipelines which is a reference to the 1ESPipelineTemplates repo (set in the resources: node at the top of the pipeline). So, providing parameters to that template will use that path within the context of that repo. Here, that context doesn't exist as this is just a file within the Installer repo with a path to a file within Installer repo. Does that make sense?

[Forgind]

@self?

[MiYanni]

See explanation above.

[Forgind]

@self?

[MiYanni]

See explanation above.

[Forgind]

Why are these all debug images?

[MiYanni]

They aren't. That's not specifying the image used. It is just the name of the job. Debug here means the configuration, specified 2 lines down: buildConfiguration: Debug

[mthalman]

@MiYanni - This has been causing failures in the Guardian analysis of all the internal CI builds.

[MiYanni]

@mthalman True. The failure is the fault of the MicroBuild signing plugin. I've attempted to add baselines that would normally resolve the build issue. However, the file it is failing on, MicroBuild/Plugins/nuget.config, changes every install, which means it cannot be baselined. Discussions have been taking place in the 1ES Pipeline Templates teams channel. Let me know if you'd like a link to that.

So far, I tested one workaround which did not seem to work with how Arcade is using the plugin. I'm currently trying the brute force method of just disabling the tooling temporarily to unblock builds. I'll have a PR soon if that is successful.

[MiYanni]

@mthalman I've merged a temporary fix to allow Installer builds of main to work properly again. I'll revert this fix next week when the Arcade updates flow in.