Add support to add ambassador circuit-breaker in seldon deployment

[anggao]

Trying to fix #1556
With this PR, users will be able to add ambassador circuit breakers from model deployment annotations.

[seldondev]

Hi @anggao. Thanks for your PR.

I'm waiting for a SeldonIO member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[axsaucedo]

/ok-to-test

[axsaucedo]

/cc @axsaucedo @cliveseldon

[seldondev]

Sat Apr 4 11:17:42 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=1

[seldondev]

Sat Apr 4 11:17:50 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=2

[axsaucedo]

/test integration

[axsaucedo]

Thanks @anggao! Looks awesome!

[seldondev]

Sat Apr 4 11:22:10 UTC 2020
The logs for [integration] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=3

[axsaucedo]

@anggao all looks good! Would you be able to add a notebook example under the examples/ folder? Similar to the examples that we have in the ambassador page in the docs? That would be very useful so other people can also use this functionality

[anggao]

@axsaucedo sure will do, will also update the doc.

[anggao]

@axsaucedo I added example please review again.

[seldondev]

Sun Apr 12 16:55:59 UTC 2020
The logs for [pr-build] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=4

[seldondev]

Sun Apr 12 16:56:03 UTC 2020
The logs for [lint] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=5

[axsaucedo]

@anggao great! Thank you for adding the example, I was thinking perhaps we could show the circuitbreaker in action in the notebook. For that basically the example maybe you coul actually set the annotations to:

"seldon.io/ambassador-circuit-breakers-max-connections":"1",

Then we could use a model such as:

import time

class Model:
     def predict(self, *args, **kwargs):
         time.sleep(20)
         return "done"

With this we could now show the result from two concurrent commands using something like

import multiprocessing
from seldon_core.seldon_client import SeldonClient

pool = mp.Pool(mp.cpu_count())

sc = SeldonClient(...)
def predict_fn(i):
    try:
        result = sc.predict(data="input")
        print(f"Successful request: {result}")
    except:
        print(f"Failed request {i}")

results = [pool.apply(predict_fn, args=(i)) for i in range(3)]

Would this be something you could do? Conscious it may take time, if you can't let me know and I can try to add further changes, just thinking if we add an example it would be good to show how it's enforced.

Alternatively actually, we could just add in the docs a reference to the Ambassador documentation where it specifies exactly what each of the ambassador annotations do - that way we'd ensure it's up to date.

[anggao]

@axsaucedo Yeah good point, in order to test this we need to make sure Ambassador is deployed as the ingress, and make sure the request go through it. I am not familiar how the python SeldonClient works, would be great if you take this over and improve the doc later, thanks!

[axsaucedo]

@anggao no worries, I added a simplified version now, perhaps we can use that? https://github.com/axsaucedo/seldon-core/blob/ambassador-circuit-breakers-example/examples/ambassador/circuit_breakers/ambassador_circuit_breakers.ipynb

One thing is that it seems in this example, the circuit breaker wasn't triggered, as both requests seem to execute as expected. The service does seem to have the ambassador annotations below but don't seem to be enforced. Did they work when you ran them?

     circuit_breakers:
      - max_connections: 1
        max_pending_requests: 1
        max_requests: 1
        max_retries: 3

[anggao]

@axsaucedo can you change the command to something like below, and see whether it works ?

seq 1 200 | xargs -n1 -P10 curl "http://localhost:5000/example"

This will run curl command 200 times with max 10 jobs in parallel.

[anggao]

@axsaucedo Any updates on this, have you tried the command above ?

[axsaucedo]

@anggao I have just tried it today, it all looks good! I will merge this and will update the notebook to use the one provided above
/approve

[axsaucedo]

/test integration

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[seldondev]

Mon Apr 27 13:02:08 UTC 2020
The logs for [integration] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=6

[seldondev]

Mon Apr 27 14:34:09 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=7

[seldondev]

Mon Apr 27 14:34:11 UTC 2020
The logs for [lint] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1661/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1661 --build=8

[anggao]

@axsaucedo great, thanks!