Build failsafes for latest/stable retagging

[ericholscher]

There are a number of classes of bugs that we have where we get stuck in build loops over and over again for builds. A few of them:

• A person retags a tag, and two of our builders each have the tag at a different commit. Each time we rebuild, they re-update the tag
• A person pushes and then updates/removes a tag, and our builders have a different version of stable that they keep re-updating and re-building in a loop
• A person has a branch named latest and we had a bug where we overwrote latest twice (once for theirs, once for ours) on each build, causing a rebuild

I believe most of these bugs have been fixed, but we should have some kind of detection or failsafe against this.

Possible solution

Have a history of previous latest/stable versions in cache that goes back 5-10 versions, with a 1 day expire date. We could then check to make sure they aren't looping (the current thing we are about to set doesn't appear in the list already?). This shouldn't take too much memory if we keep a low expires date, but should at least stop a looping effect.

We should probably do something similar to user notifications and other places where a loop might be possible and really bad. Perhaps a cache of times an email address has been notified, or a version built, and cap it in some ongoing fashion?

Refs #3180

[agjohnson]

Sounds like we need to start with replicating some of the failure cases first, so we know what we're testing/changing.

[stsewd]

A person retags a tag, and two of our builders each have the tag at a different commit. Each time we rebuild, they re-update the tag

A person pushes and then updates/removes a tag, and our builders have a different version of stable that they keep re-updating and re-building in a loop

I think these two were solved with #4901 (we re-sync the versions before triggering a build), and #4876 also could help here.

A person has a branch named latest and we had a bug where we overwrote latest twice (once for theirs, once for ours) on each build, causing a rebuild

I believe that we already fix that in #4733 and/or #3913

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

As my comment in #3513 (comment) I'm closing this issue.

Also, we are looking further for have aliases for latest and stable :)