Skip to content

Latest commit

 

History

History
346 lines (280 loc) · 14.3 KB

PROCESSES.md

File metadata and controls

346 lines (280 loc) · 14.3 KB
Raw

Processes, release procedures etc

This repository includes multiple projects at multiple versions, some of which depend on each other. That makes things tricky, and leads to rather more complicated procedures than a simple "single package" repository. Please read this guide carefully to understand how releases happen, and why they happen that way.

The API catalog

Only the source within the apis directory is published, although not every project within that directory is published. (There are test projects and tools alongside the production code.) Nothing in the tools directory is published as a package.

Most project files under apis are at least partially generated. Metadata used for generation is in apis.json - the API catalog file. There's an entry for each API, containing:

  • The kind of API (grpc, rest, other)
  • The version number
  • A product title and home page for documentation
  • Dependencies
  • Additional test project dependencies
  • Target framework versions
  • Package description for NuGet
  • Tags for NuGet (in addition to default ones)
  • Proto path within the googleapis repository
  • Service YAML file

The catalog is used to generate project files and also during the release process described below. Running the project generator is very simple: from the root directory, in a bash shell, run

./generateprojects.sh

The CI systems run this before building, to ensure that the project files are in a stable state.

Generating the project files allows for broad changes (such as adding Source Link support) to be made very simply, just by changing the generator. Modifying every project file by hand simply doesn't scale.

However, sometimes manual editing of project files is required. The project generator supports this by assuming it "owns":

  • The first PropertyGroup element (for general properties)
  • The first ItemGroup element (for dependencies)
  • The ItemGroup element for Source Link (with a label of "dotnet pack instructions")
  • The common import to only attempt to build desktop assemblies on Windows

Any other elements are left as they are - so if you wish to add an ItemGroup such as for file grouping, just add it anywhere after the generated one, and it should do the right thing.

Additionally, for each API, the project generator adds all projects under the API directory (and project references) to the solution for that API. It will create project files from scratch as well - so when adding a new package from autogenerated API sources, the simplest approach is to copy the source files, modify the API catalog, and run the project generator. The project files and solution file will be generated and should immediately be usable.

Versioning

All releasable packages follow Semantic Versioning. Non-releasable code (tests, tools, analyzers) are not versioned. The precise meaning of a breaking change (dictating version number increments) is out of the scope of this document. (Jon is writing a blog post about this topic and will link to it when it is published.)

The version number in the API catalog (and therefore in project files) is one of:

  • The mostly recently released package version
  • The version that's about to be released (see below)
  • A prerelease with a suffix of 00 to indicate that the next release should be the first prerelease for that major/minor version.

The last of these cases can happen if either an existing version has gone to GA and new features have been merged since (to ensure that we don't add features within a minor GA version) or for brand-new APIs that haven't been released at all. The tooling for tagging is aware of this convention, and doesn't create tags for alpha00 or beta00 suffixes.

Typically version numbers should be changed in a commit which does nothing else, for clarity. Include both the apis.json change and the project file changes that occur when the project generator has been run, in the same commit.

Dependencies

Dependencies in the API catalog are specified as properties where the property name is the package name and the value is the version number.

If the version number is set to "project", then a project reference is used. If project X has a production dependency on project Y, then both X and Y must be released together, to avoid misaligned versions. The tagging tool enforces this.

If the version number is set to "default", then the version number is determined by the project generator, to keep these dependencies in sync for all appropriate packages.

Two project types gain dependencies automatically if they're not specified:

  • "grpc" projects always have dependencies on Google.Api.Gax.Grpc and Grpc.Core
  • "rest" projects always have a dependency on Google.Api.Gax.Rest

Note that as of 2024-03-18, even GA versions can use default dependencies. This means that when creating a patch release (which is relatively rare) the previous version's dependencies should be checked in NuGet and the same versions (or patch-level only changes) should be explicitly listed. This is checked within the project generator.

Releasing

Releasing consists of these steps:

  • Updating the version number in GitHub (via standard pull requests), as well as the version history.
  • Creating a release tag and GitHub release
  • Building and testing
  • Pushing the package to nuget.org
  • Updating the documentation in GitHub (in the gh-pages branch)

The last three of these are typically performed by a Kokoro release job.

Detailed steps

Much of this work is automated by the prepare-release.sh script, but allows for intervention at any point. This documentation assumes the use of prepare-release.sh, but that's just for simplicity. Everything can be done manually.

Note that only the set-version command for prepare-release needs the package ID to be specified. All other commands work on the basis of "whatever versions have been changed since the latest commit on GitHub are the ones we want to release".

Creating a GitHub Personal Access Token

The release process uses a GitHub Personal Access Token to push to the repository and create a pull request. In order to reduce the impact of any access tokens being leaked, we use fine-grained personal access tokens. These can only authorize a single organization or user. We therefore push the release branch to the main fork, so we can create the pull request with the same token.

Create a new access token using the GitHub token creation page. The token should have:

  • A suitable description and name
  • A short-ish expiry date (30 days is fine if you're going to use it multiple times, storing it securely; for one-off use, set it to the shortest available expiry)
  • A resource owner of googleapis
  • Access to only the google-cloud-dotnet repository
  • Repository permissions for "Contents" and "Pull Requests", both set to "Read and write". (There'll be a read-only metadata permission as well, which is automatically included.)

Once you have created the access token, set the GITHUB_ACCESS_TOKEN environment variable to the token value in the shell you're running prepare-release.sh from.

Create the release PR

  1. Make sure the main branch is up-to-date (as that's what prepare-release.sh uses to determine the current versions) and create a new branch from that. If you want to use the compare option later, run git fetch --all --tags -f as well to make sure you have all the latest tags.

  2. Run ./prepare-release.sh set-version <api> <new-version>, e.g. prepare-release.sh set-version Google.Cloud.Speech.V1 2.0.0-beta03. This updates apis.json and regenerates the project and metadata files. The output includes the old version and the new version, so you can check this is really what you meant to do.

    As an alternative to set-version, you can run ./prepare-release.sh increment-version <api> which will increment the minor version for GA libraries (e.g. 3.1.0 to 3.2.0) or the prerelease version of non-GA libraries (e.g. 3.0.0-beta01 to 3.0.0-beta02). This just avoids you having to look up the current version number if you don't already know it. If you want to look up the version number anyway, use ./prepare-release.sh show-version <api>.

    If the API you're releasing is part of a package group (multiple packages that need to be released together), the increment-version command will automatically increment all of them.

  3. If you want to perform an API surface comparison, run ./prepare-release.sh compare. This will build the code locally, and compare the previously released NuGet packages with the current source code, showing you what's changed. Check there's nothing unexpected. Note that this will only show changes to the API surface; internal changes (even if they change behavior) will not be shown.

  4. Run ./prepare-release.sh update-history. This will update the version history files of all changed APIs, based on the commits that touched the relevant directories. The command displays any additions that it makes to the version history, along with the name of the history file for each API.

  5. If you're only releasing a single package, run ./prepare-release.sh commit. This will commit all the current changes, with a message taken from the version history for the package. Use git commit --amend to change the commit message if you need to.

    If you're releasing a package group, use ./prepare-release.sh commit-group. Alternatively, create the commit manually, including one line per package of the form - Release XYZ version ABC.

    Note: If you're using the Dockerfile described in ENVIRONMENT.md, please read the notes there around commiting to git "inside" vs "outside" the container.

  6. Run ./prepare-release.sh push to push the current branch and create a pull request with the autorelease: pending and automerge: exact labels. If the RELEASE_PR_ASSIGNEE environment variable is set, the PR is created with that assignee. Note that this command checks that there are no project references from APIs being released now to APIs that aren't being released. Without this check, it's possible for a released version to depend on unreleased changes.

Sample session when releasing Google.Cloud.Speech.V1:

$ git checkout main
$ git pull upstream main
$ git fetch --all --tags -f
$ git checkout -b release-speech
$ ./prepare-release.sh set-version Google.Cloud.Speech.V1 2.0.0-beta03
$ ./prepare-release.sh compare
$ ./prepare-release.sh update-history
$ ./prepare-release.sh commit
$ export GITHUB_ACCESS_TOKEN=...
$ ./prepare-release.sh push

Equivalent process using increment-version, assuming the current version is 2.0.0-beta02:

$ git checkout main
$ git pull upstream main
$ git fetch --all --tags -f
$ git checkout -b release-speech
$ ./prepare-release.sh increment-version Google.Cloud.Speech.V1
$ ./prepare-release.sh compare
$ ./prepare-release.sh update-history
$ ./prepare-release.sh commit
$ export GITHUB_ACCESS_TOKEN=...
$ ./prepare-release.sh push

Building and publishing the release

Once the pull request is merged, the commit will be tagged automatically, and a release will be created. A Kokoro release build will then execute automatically.

The Kokoro build will:

  • Fetch the commit
  • See which tags are present for that commit (to work out which packages need releasing)
  • For all those packages:
    • Build
    • Run unit tests
    • Run integration/snippet tests
    • Build documentation
    • Push packages to nuget.org
    • Push documentation to GitHub packages
    • Push documentation to googelapis.dev

Batch releasing

Occasionally (e.g. after regenerating all libraries with a generator change, or after a GAX release) it can be useful to perform many releases in quick succession.

The release manager tool has a batch release command for this purpose. This is a relatively advanced usage scenario, and is documented alongside the source code.

Sample configurations are in tools/BatchReleaseConfigurations.

Turning down a library

When an API has been deprecated, our process for turning down the corresponding library is:

  • Release a new version with a modified docs/index.md file to indicate the deprecation. This should also be indicated in the release notes. See this PR as an example.
  • Mark all published versions as deprecated and delisted in NuGet. The deprecation step can easily be done manually as the NuGet web site allows marking all versions as deprecated in one go. For delisting, either delist each version manually, or use the release manager delist-package command to delist all versions.
  • Create a PR to remove the API from the API catalog, delete the source code, and regenerate projects (which will update Renovate and the README). See this PR as an example.

Updating the .NET SDK

Periodically, we need to update the version of the .NET SDK we use for building. This might be to gain access to new C# features, or due to security patches being available. This must be done carefully in order to avoid other processes breaking. Only start this process when you are confident that all the steps below can be completed in a reasonable timeframe.

  1. Update internal CI systems to include the new SDK.
  2. Create a PR with the following changes:
    1. Update global.json to specify the .NET release version required.
    2. Edit owl-bot-post-processor/Dockerfile to change the base image.
    3. For new major SDK versions, update GitHub actions
  3. Merge the above PR.
  4. Wait for a new image to be created automatically in gcr.io/cloud-devrel-public-resources/owlbot-dotnet
  5. Create a PR to edit .github/.OwlBot.lock.yaml, specifying the new image hash
  6. Merge the above PR.
  7. If there are any outstanding OwlBot PRs, for each PR:
    1. Fetch the PR locally
    2. Rebase it to main (which now has the updated lockfile)
    3. Force-push back to the PR branch
    4. Add the owlbot:run label