Retry storage access on internal errors

[fische]

This revision adds a retryBucket struct that wraps the bucket interface in order to implement a retry logic on top of it. We go through this logic only on internal errors.

We are implementing this following an increase in the number of internal errors coming from reads on GCP buckets. The only suggested solution is to just retry when those errors occur: googleapis/google-cloud-go#784

[peterebden]

Looks good but this doesn't handle the case where mettle workers go directly to storage (probably that wants a retryable wrapper around the 'clients' in this package too).

It also won't retry if you get an initial reader or writer successfully but it fails later. Do we know if that happens or if it's generally upfront that it fails?

[peterebden]

You might also want to rebase on master - #181 eventually got a green build out of golangci.

[fische]

Looks good but this doesn't handle the case where mettle workers go directly to storage (probably that wants a retryable wrapper around the 'clients' in this package too).

Do you mean when we use the elanClient instead of the remoteClient implementation? If that's what you mean, it's already using this wrapper as New calls createServer.

It also won't retry if you get an initial reader or writer successfully but it fails later. Do we know if that happens or if it's generally upfront that it fails?

That's a very good point. There's no indication that this is only happening when initialising the reader/writers, so I'll add retries for that as well.

[fische]

Abandoning this PR, as we can just use the retry logic from the GCS client, we just need to update the storage package for that.