Being spammed by endless failed build loop

[dlech]

Details

• Project URL: https://readthedocs.org/projects/grx/
• Build URL (if applicable): https://readthedocs.org/projects/grx/builds/6159547/ (and 250 more so far!)
• Read the Docs username (if applicable): dlech

Expected Result

I should not be getting failed build emails every 7 minutes.

Build log should actually show a log instead of only an error message.

Actual Result

readthedocs.org was emailing me all night long.

I've tried disabling, wiping and clearing all of the related branches/tags but I am still getting error messages for "stable" (the readthedocs.org special feature, not a branch or a tag) even though I disabled, wiped and cleared "stable".

The build "log" says: "An unexpected error occurred" or sometimes "Version locked, retrying in 5 minutes."

I've tried everything I can think of, but I can't seem to break this loop of endless retries.

[RichardLitt]

@dlech Sorry about this. Is there any way you could set up a temporary filter to ignore those emails in the meantime?

[RichardLitt]

I'm going to assign this to priority: high, because spamming people is a little worse than just having a bug, I think.

[ericholscher]

Should be fixed. Did you re-tag a version by chance?

[ericholscher]

I think it's from having two tags with <string>/3.0.0 and a couple of our builders each remarking the each as the "latest"

[ericholscher]

Believe the logic that stops new stable builds was also not getting triggered, since it was doing a stream of retry's, instead of new builds :/

[ericholscher]

Going to leave this open to try and fix this logical fail though! :)

[dlech]

Thanks! It looks like the emails stopped a few hours ago.

[agjohnson]

@ericholscher do you know if this would have been fixed by the changes to stable last week?

[ericholscher]

@agjohnson no, this is basically the same bug, but for any tag -- and caused by someone re-tagging and two of our builders having different commits for the same tag, and then bouncing back and forth :)

[agjohnson]

I vaguely remember addressing this. @ericholscher do you know if this is still a valid bug?

[ericholscher]

We fixed the root cause of this specific bug, but we should really have some kind of failsafe on our notifications and retagging of latest/stable, to stop the general case. We could open a new ticket for that though.