Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the entire Azure DevOps build system #34984

Merged
merged 84 commits into from
Sep 25, 2024
Merged

Conversation

DHowett
Copy link
Member

@DHowett DHowett commented Sep 20, 2024

This pull request rewrites the entire Azure DevOps build system.

The guiding principles behind this rewrite are:

  • No pipeline definitions should contain steps (or tasks) directly.
  • All jobs should be in template files.
  • Any set of steps that is reused across multiple jobs must be in
    template files.
  • All artifact names can be customized (via a property called
    artifactStem on all templates that produce or consume artifacts).
  • No compilation happens outside of the "Build" phase, to consolidate
    the production and indexing of PDBs.
  • All step and job templates are named with step or job first,
    which disambiguates them in the templates directory.
  • Most jobs can be run on different pools, so that we can put
    expensive jobs on expensive build agents and cheap jobs on cheap
    build agents. Some jobs handle pool selection on their own, however.

Our original build pipelines used the VSBuild task all over the
place.
This resulted in PowerToys being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where ci.yml consumed jobs and steps templates... but when
release.yml was added, all of that went out the window.

It's the same story as Terminal (microsoft/terminal#15808).

The new pipelines are consistent and focus on a small, well-defined set
of jobs:

  • job-build-project
    • This is the big one!
    • Takes a list of build configurations and platforms.
    • Produces an artifact named build-PLATFORM-CONFIG for the entire
      matrix of possibilities.
    • Builds all of the installers.
    • Optionally signs the output (all of the output).
    • Admittedly has a lot going on.
  • job-test-project
    • Takes one build config and one platform.
    • Consumes build-PLATFORM-CONFIG
    • Selects its own pools (hardcoded) because it knows about
      architectures and must choose the right agent arch.
    • Runs tests (directly on the build agent).
  • job-publish-symbols-using-symbolrequestprod-api
    • Consumes **/*.pdb from all prior build phases.
    • Uploads all PDBs in one artifact to Azure DevOps
    • Uses Microsoft's internal symbol publication REST API to submit
      stripped symbols to MSDL for public consumption.

Finally, this pull request has some additional benefits:

  • Symbols are published to the private and public feeds at the same
    time, in the same step. They should be available in the public symbol
    server for public folks to debug against!
  • We have all the underpinnings necessary to run tests on ARM64 build
    agents.
  • All dotnet and nuget versioning is consolidated into a small set of
    step templates.
  • This will provide a great place for us to handle versioning changes
    later, since all versioning happens in one place.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

@DHowett DHowett marked this pull request as ready for review September 24, 2024 21:30
@DHowett DHowett changed the title WIP: Rewrite the entire Azure DevOps build system Rewrite the entire Azure DevOps build system Sep 24, 2024
@DHowett
Copy link
Member Author

DHowett commented Sep 24, 2024

I've moved all the pipelines to the v2/ subdirectory, and will change the build definitions to match once this lands. This will cause some indigestion for older versions and some inflight PRs until they merge main.

@DHowett
Copy link
Member Author

DHowett commented Sep 24, 2024

I am still experimenting with the arm64 test runner - I would love to see it work but I understand if it can't work (and I have just one commit to revert if it doesn't.)

@crutkas
Copy link
Member

crutkas commented Sep 25, 2024

🔥🔥🔥🔥🔥

@crutkas crutkas added the Needs-Review This Pull Request awaits the review of a maintainer. label Sep 25, 2024
@DHowett
Copy link
Member Author

DHowett commented Sep 25, 2024

I am investigating an issue where the signatures on the nuget packaged DLLs are not present. Step 1 for tomorrow is to determine whether this is true in the original pipeline as well.

@DHowett
Copy link
Member Author

DHowett commented Sep 25, 2024

I am investigating an issue where the signatures on the nuget packaged DLLs are not present. Step 1 for tomorrow is to determine whether this is true in the original pipeline as well.

THIS IS NOT A REGRESSION

Copy link
Collaborator

@jaimecbernardo jaimecbernardo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for the changes!

${{ else }}:
OutputBuildPlatform: ${{ platform }}
variables:
# Azure DevOps abhors a vacuum
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤣

@crutkas crutkas merged commit ad1f204 into main Sep 25, 2024
21 checks passed
Comment on lines -29 to -30
variables:
EnablePipelineCache: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont' see this in the new ci.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs-Review This Pull Request awaits the review of a maintainer.
Projects
Development

Successfully merging this pull request may close these issues.

5 participants