intermittent UNIMPLEMENTED error from ambassador

[ryandawsonuk]

Spotted this in the helm_examples notebook but have also seen in the E2E tests. Was seeing:

_Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNIMPLEMENTED
    details = ""
    debug_error_string = "{"created":"@1553618148.143629760","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"","grpc_status":12}"

Might especially affect the A/B test scenario but not sure. Was encountered when testing #445 but it isn't related to that change as has been seen separately.

[ryandawsonuk]

Seems to especially affect first request so could be related to emissary-ingress/emissary#504
Also related to grpc/grpc#16515 as 503 is unavailable and it seems this is the behaviour for a network connectivity failure.

Looking at grpc/grpc#16515 (comment) perhaps the way forward for this is https://pypi.org/project/retrying/

[ryandawsonuk]

Seem to be able to resolve this for tests using 656ac5b#diff-3119fc203a07a67876b8d74d342faceeR44 - have only applied that in a particular branch for now

[ryandawsonuk]

I think upgrading to latest ambassador (through #480) also resolves this for the notebooks. I was seeing this intermittently in notebooks and haven't seen it in that branch with the latest ambassador

[ryandawsonuk]

Turns out the issue is present in 0.53.1 but can be worked around by setting ambassador to run as root.

Will check again in 0.60.0 - emissary-ingress/emissary#504

[ryandawsonuk]

Actually issue still present even when running as root. I managed to recreate the ambassador grpc problem from the helm_examples.ipynb. This time though I didn't uninstall and reinstall ambassador. I just followed the notebook and for the AB test scenario (at which point ambassador has been there for a while) I saw was the grpc calls reporting as failed for a good minute while REST requests worked. It seems the issue actually affects the initial period when a new instance comes online, specifically for A/B tests. Oddly, ambassador logs were reporting that the requests were going through successfully and there were no errors in other logs but the response was clearly a failure:

Eventually grpc requests start working but it can take a whole minute. The failures always seems to be when using ambassador with single namespace scope.

We can work around this by downgrading to 0.50.0, even though that means putting references to the ambassador API back down to v0.

[damitkwr]

Is this issue still present in Ambassador v73.0?

[ukclivecox]

Its not been confirmed or closed by Ambassador : see emissary-ingress/emissary#1587

[axsaucedo]

+1

[lennon310]

We are using ambassador 0.73.0 (quay.io/datawire/ambassador:0.73.0), and we are having this gRPC issue. May I have an update on how to resolving this?

[ukclivecox]

We need to determine with Ambassador community if this is a Seldon issue or Ambasador issue. If Ambassador we need to evaluate if we continue to support Ambassador long term.

[seldondev]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[ukclivecox]

This issue should be fixed for grpc. There is still the issue of envoy updates not always happening immediately in an Ambassador update which means e2e tests can be flaky.