Identify an automated revert path for infrastructure releases

[srtalbot]

Description:

This card was from an incident that led to a temporary delay in sending form responses via Notify.

Currently TF plan is run if it's a release-please-- prefixed branch. We could update the trigger on this to instead only run if the version.txt file has been changed in the PR and its target is the develop branch. Since Release Please always updates this file, TF plans would keep running as expected.

TF apply is set to only trigger when a new release is published. If this was instead updated to trigger when the version.txt file was updated, we could then update the checkout step to use this tagged version instead.

Benefits:
Release Please PR merges and revert PR merges would behave the same way (running TF plan and TF apply) allowing for easier rollbacks.

Potential gotchas:
We may end up in a race-condition with step 2 since Release Please is responsible for publishing the release. We'd likely have to add safeguards into the TF apply to wait for the release to exist.

[patheard]

The plan is to make the following changes:

Terraform plan production

1. Update the checkout step of the terragrunt-plan-production.yml workflow to check if the version.txt tag already exists. If it does, this is a revert PR and that tag should be checked out. Otherwise it is a normal release flow.

Terraform apply production

1. Update the trigger on the terragrunt-apply-production.yml workflow to run when the version.txt file is changed.

2. The checkout step will be updated to wait until the expected tag exists and fail after a reasonable polling timeout. Once it exists, this tagged version will be checked out. This wait is required because there are now two potential paths for this workflow to execute:

   1. During a normal release-please PR merge: this merge will trigger the release_generator.yml workflow, which will create the tag. The reason a release dispatch event is not being used is because this would cause a double invoke of the terragrunt-apply-production.yml workflow: once for the release dispatch and once for the version.txt file change.
   2. When a revert PR has been merged: this merge will only trigger the terragrunt-apply-production.yml workflow and the tag will already exist.

Considerations

Testing will need to be done to see how release-please behaves once a revert PR has been merged. It does provide a Release-As commit message hook that should allow us to resume normal semantic versioning.

Alternatively, it would also be possible to delete the "bad" tag it created and have it recreate that version. This is the approach suggested by one of the maintainers:

• How fix a release that went out as major bump googleapis/release-please#1594

[srtalbot]

Will look to merge PR in draft after testing, after Tuesday next week and app router.

[patheard]

This has now complete, along with a change to enforce CODEOWNER review on the version.txt file to prevent accidental prod releases.