Ambassador config for rolling updates

[ukclivecox]

When doing a rolling update of a model the kubernetes endpoint might change and requests passing through ambassador should handle the new ip-address without service-504 errors. One suggestion from Ambassador Slack channel is to use an Ambassador config like:

     timeout_ms: 0
     envoy_override:
       retry_policy:
         retry_on: connect-failure
         num_retries: 4

This needs to be tested and if works we should decide if its added by default.

[stefansedich]

@cliveseldon this is something I just ran into myself, is there an easy way to set this for my testing or is it something that needs to be done within seldon itself?

[stefansedich]

Ignore me @cliveseldon I see where it is now!

[ukclivecox]

Yes, you would need to add to update the Ambassador annotation in the created Service. We'd love to hear your feedback if it helps you so we can increase priority to do this.

[stefansedich]

Big +1 from me!

…

On Sun, Feb 17, 2019, 12:29 AM cliveseldon ***@***.***> wrote: Yes, you would need to add to update the Ambassador annotation in the created Service. We'd love to hear your feedback if it helps you so we can increase priority to do this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIqVAaXxDsaIMtnjayhmtDWRD1Xuvbfks5vORLpgaJpZM4YZbw9> .

[ukclivecox]

@stefansedich Can you confirm your use of these Ambassador settings solved the issue for you?

[stefansedich]

@cliveseldon I have not had a chance to test it out yet, will let you know how it goes when I get some time to do it.

[ukclivecox]

It looks like for Ambassador 0.5 release envoy_override has been removed for now so this functionality is only available for 0.4: See emissary-ingress/emissary#1127

[ukclivecox]

Might be handled soon in 0.5 emissary-ingress/emissary#1358

[ukclivecox]

Ambassador has this now emissary-ingress/emissary#1127

[ukclivecox]

Config added by default now in master with latest seldon-operator