Activator: Retry on Get Revision error

[akyyy]

/area API
/area autoscale

/kind bug

Expected Behavior

Get revision calls have transient errors. #1558. Activator should deal with that.

Actual Behavior

Activator uses Get revision api as one time shot, sometimes it fails with this error in the log:
Unable to get revision: Get https://10.35.240.1:443/apis/serving.knative.dev/v1alpha1/namespaces/default/revisions/configuration-example-00001: unexpected EOF
In this case, a potential reason is master is down.

Steps to Reproduce the Problem

This usually doesn't happen.

[mattmoor]

We should consider replacing the activator's direct client usage with something informer based. This is only peripherally related to this issue, but is probably how the only real action I'd take on our side to fix this.

The activator and activation in general is at the mercy of the availability of the K8s control plane (I've been meaning to write up an issue on this for a while).

[mattmoor]

cc @dgerd

[dgerd]

/assign @dgerd

I am going to take a stab at replacing the client usage with an informer.

[dgerd]

/unassign

I have not made progress on this. Going to unassign for now.

[vvraskin]

/assign
I'm going to follow up on this one once the #2653 gets in.

fyi @dgerd

[dgerd]

Thanks for picking this up. Let me know when you need another look at #2653.

[mattmoor]

@vvraskin I'm going to put this into 0.4 then

/milestone Serving 0.4

[mattmoor]

Moving the work left to 0.5 so that we can bake alongside a new e2e test.

[yanweiguo]

Hi @vvraskin, do you have any update for this one? Thanks.

[vvraskin]

Unfortunately I don't have cycles for this one in the next weeks.
/unassign

[markusthoemmes]

Gonna pick this up

/assign