Add a few retries around failure points in release scripting

[Verolop]

What would you like to be added:

Adding some retries around failure points in release scripting before failing completely. Eg:

• docker pull commands on the code for Mock Release (Issue Implement clone, commit and push of release notes draft to user's fork #1102 )
• "Step Initial commit of release automation tooling. #1: PUSH GIT OBJECTS (11/14)" (Issue gcbmgr release frequently fails at "push git objects" #933 )
• audit past year's "HACK" and "revert" commits on anago/gcbmgr/lib shell to other points which would benefit from a basic sleep/retry loop

Why is this needed:

In case of failure, often we just need to hack workaround and attempt to re-run the release process, which is very time and resource consuming. Adding retries will allow these attempts to take place on the same run.

[justaugustus]

/assign @Verolop @cpanato
ref: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1591814834391800?thread_ts=1591794999.379400&cid=CJH2GBF7Y

[justaugustus]

/remove-priority important-soon
/priority critical-urgent

[tpepper]

Simple retries might help us limp along.

But long term we likely must run our own container image registry and mirror external content. Requiring the internet to be consistent/coherent in order to build is always going to be problematic. Failing to build because we can't docker pull docker.io/library/debian:stretch-slim should not happen.

[Verolop]

sounds good, I agree!

[tpepper]

@Verolop we've just discussed that there are a number of issues that are relatively similar to this, so I've tweaked the description slightly to cover the more abstracted case. There's max half a dozen points where a minimal additional retry loop in the shell script could make us much more likely to survive these random transient failures and save tonnes of release time and effort.

[tpepper]

(ie: rather than close a bunch of issues and create new one...just re-using/re-focusing this one for broad impact)

[saschagrunert]

An idea which came into my mind: What if we add a krel subcommand for pushing the git objects? Seems fairly straight forward and we could remove the bash bits from anago. Then I'd like to enhance the logging via logrus and maybe add some pre-checks: For example making the call fail only in some certain cases and assume that "everything is ok" if the tag is already present remotely. WDYT?

[tpepper]

An idea which came into my mind: What if we add a krel subcommand for pushing the git objects? Seems fairly straight forward and we could remove the bash bits from anago. Then I'd like to enhance the logging via logrus and maybe add some pre-checks: For example making the call fail only in some certain cases and assume that "everything is ok" if the tag is already present remotely. WDYT?

This is the type of decomposition we need. The anago bash bits doing that push can be removed to instead have anago call a more robust pusher.

[tpepper]

Related on the topic of fail/retry/continue resilience:
kubernetes/test-infra#18808

[Verolop]

Does it still make sense to introduce the retries at this point, or should we just go ahead with @saschagrunert 's idea?

[tpepper]

We definitely need some retries still in anago too.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[puerco]

/remove-lifecycle stale

This was mostly addressed in #1595

unless there are more suggestions and/or comments I think we can close this one

[cpanato]

Agree, if more things come up we can create a new issue

/close

[k8s-ci-robot]

@cpanato: Closing this issue.

In response to this:

Agree, if more things come up we can create a new issue

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.