Using tags to initiate building release binaries is problematic

[KristofferC]

Right now, a new tag will trigger building of new releases. However, this means that we will not know if there is some issue with the release until after the tag is made. It is non-ideal to delete tags since they people who have already pulled the tag will not get the updated ones.

So it would be better to have some other workflow where we only create the tag once the binaries are built.

Perhaps some people have suggestions of good workflows.

[DilumAluthge]

Right now, a new tag will trigger building of new releases.

We do build the binaries on every push to the release-1.x branch. This will probably catch most issues. So the right workflow might be:

1. Merge the backports PR into the release-1.x branch.
2. Wait for the CI jobs to finish.
3. Check the CI jobs on the latest commit on the release-1.x branch.
4. Only if the CI jobs passed, bump the VERSION and proceed with making a tag.

As an example, we can look at: https://buildkite.com/julialang/julia-release-1-dot-9/builds?branch=release-1.9

On that page, if we look at the build titled Merge pull request #47602 from JuliaLang/backports-release-1.9 (Build #23), we see that the upload_macOS jobs failed on that build. So we would have caught the macOS codesigning/notarization issue (#48019) if we'd waited for that build to succeed before we proceeded with making a tag.

[DilumAluthge]

@staticfloat Let's discuss this further during our next #ci-dev call.

For anyone interested in joining the call, the details are in the #ci-dev channel on Slack.

[KristofferC]

It becomes quite a number of steps now where you have to wait for CI with that workflow.

1. Wait for backport PR to finish CI.
2. Wait for merged backport PR to finish CI.
3. Bump VERSION, wait for CI to merge.
4. Make tag, wait for CI to make binaries.

Ideally, it should be enough to just bump the VERSION as the last commit on the backport branch and when that is merged the rest happens automatically.

[DilumAluthge]

I wonder if we could have a GitHub Action to automate most of this?

So then, the human would do the following steps:

1. Put a VERSION-bumping commit as the last commit on the backports branch.
2. Wait for CI to pass on the backports PR.
3. Merge the backports PR.

And then the GitHub Action would automate the following steps:

1. Wait for CI to pass on the release-1.* branch.
2. Once CI passes, automatically create the new tag.
3. Wait for CI to pass on the tag.

[DilumAluthge]

If that sounds like the right approach, I can work on an implementation.

[DilumAluthge]

I have a much easier idea. @staticfloat Let's just start running the upload_ steps on the backports-release-* PRs. That way, the CI on the backport PR will run exactly the same commands as the CI on a tag would.

So a human would only need to do the following:

1. Put a VERSION-bumping commit as the last commit on the backports PR.
2. Wait for CI to pass on the backports PR.
3. Merge the backports PR.
4. Immediately create the tag from the latest commit on the release-*

There's no need to wait after step 3, because if the backport PR passes CI, we know that the tag will also pass CI.

@staticfloat To which S3 bucket should I upload the binaries that are built on backports PRs?