Move from a custom tornado server to a seldon-core server for serving the model #36

ankushagarwal · 2018-03-09T00:52:56Z

seldon-core allows wrapping arbitrary models into a REST/GRPC API. We can leverage that feature instead of writing our own boilerplate code for an API server

Related to #11

This change is

* Move from a custom tornado server to a seldon-core model Related to #11

ankushagarwal · 2018-03-09T00:53:09Z

/cc @jlewi

jimexist · 2018-03-09T01:24:10Z

github_issue_summarization/notebooks/requirements.txt

@@ -0,0 +1,11 @@
+numpy


if you don't mind please use pipenv

Wrapping a seldon-core microservice requires a requirements.txt file: https://github.com/SeldonIO/seldon-core/blob/master/docs/wrappers/python.md#create-a-model-folder

@jimexist might want to open an issue in kubeflow/kubeflow or Seldon to follow up with them. @cliveseldon

jlewi

This is great. So glad we were able to get Seldon integrated.

jlewi · 2018-03-09T17:54:14Z

github_issue_summarization/notebooks/requirements.txt

@@ -0,0 +1,11 @@
+numpy


@jimexist might want to open an issue in kubeflow/kubeflow or Seldon to follow up with them. @cliveseldon

jlewi · 2018-03-09T17:55:20Z

github_issue_summarization/serving_the_model.md


-Start the server using `python server.py --port=8888`.
+> The model is written in Keras and when exported as a TensorFlow model seems to be incompatible with TensorFlow Serving. So we're using seldon-core to serve this model since seldon-core allows you to serve any arbitrary model. More details [here](https://github.com/kubeflow/examples/issues/11#issuecomment-371005885).


Might be good to open up an issue to track models that TFServing doesn't work with so we can loop in various folks e.g. from TFServing team. Its unfortunate that we can't use TFServing.

jlewi · 2018-03-09T17:56:56Z

github_issue_summarization/serving_the_model.md


-## Sample request
+Ensure that you have the following files from the [training](training_the_model.md) step in your `notebooks` directory:


Why notebooks directory? What if we trained the model by running a TFJob and the model is stored on GCS/S3?

I guess if they ran the TFJOb on GCS they could just do gsutil to copy from the bucket to their pod.

jlewi · 2018-03-09T18:00:42Z

github_issue_summarization/serving_the_model.md

+cd into the notebooks directory and run the following docker command. This will create a build/ directory.
+
+```
+cd notebooks/


Does Seldon's model wrapper support reading the model from GCS/S3?

Could we run the python wrapper as a job on K8s? Could we create a ksonnet template to make it easy to launch such a job.

I don't think we should try to address either of those issues in this PR. But might be good to open up issues to track them.

/cc @cliveseldon @gsunner @ahousley

Does Seldon's model wrapper support reading the model from GCS/S3?

Not that I am aware of. Looking at the instructions, it seems to require all the files in a single folder which contains the code and the model to build the serving image.

Created an issue to track the feature for serving a seldon-core model: kubeflow/kubeflow#405

Looks like there is a seldon component in kubeflow. In the next PR, I'll try to deploy this image on k8s using this component

You can specify any persistent volume in definition of the SeldonDeployment in the componentSpec as its a standard Kubernetes PodTemplateSpec. So you can load your parameters from their rather than package them as part of the model image if you wish. The kubeflow-seldon example shows an example using an NFS volume to load the trained model parameters.

jlewi · 2018-03-09T18:06:46Z

github_issue_summarization/serving_the_model.md

+./build_image.sh
+```
+
+Now you should see an image named `seldonio/issuesummarization:0.1` in your docker images. To test the model, you can run it locally using


Can we tag it into a repo like GCR/DockerHub/Quay? We'll need that to deploy the model on K8s.

Updated to use a GCR tag.

k8s-ci-robot · 2018-03-09T18:07:35Z

@jlewi: GitHub didn't allow me to request PR reviews from the following users: ahousley.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Does Seldon's model wrapper support reading the model from GCS/S3?

Could we run the python wrapper as a job on K8s? Could we create a ksonnet template to make it easy to launch such a job.

I don't think we should try to address either of those issues in this PR. But might be good to open up issues to track them.

/cc @cliveseldon @gsunner @ahousley

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jlewi · 2018-03-09T22:35:13Z

/lgtm
/approval

k8s-ci-robot · 2018-03-09T22:35:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Create a end-to-end kubeflow example using seq2seq model (4/n)

5021ea8

* Move from a custom tornado server to a seldon-core model Related to #11

k8s-ci-robot requested review from jimexist and zjj2wry March 9, 2018 00:53

k8s-ci-robot added the size/L label Mar 9, 2018

k8s-ci-robot requested a review from jlewi March 9, 2018 00:53

jimexist reviewed Mar 9, 2018

View reviewed changes

ankushagarwal changed the title ~~Create a end-to-end kubeflow example using seq2seq model (4/n)~~ Move from a custom tornado server to a seldon-core server for serving the model Mar 9, 2018

jlewi reviewed Mar 9, 2018

View reviewed changes

k8s-ci-robot requested review from ukclivecox and gsunner March 9, 2018 18:07

Update to use gcr.io registry for serving image

f937733

k8s-ci-robot assigned jlewi Mar 9, 2018

k8s-ci-robot added the lgtm label Mar 9, 2018

k8s-ci-robot added the approved label Mar 9, 2018

k8s-ci-robot merged commit d1a2adf into kubeflow:master Mar 9, 2018

ankushagarwal deleted the issue_summarization_serving branch March 9, 2018 23:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move from a custom tornado server to a seldon-core server for serving the model #36

Move from a custom tornado server to a seldon-core server for serving the model #36

ankushagarwal commented Mar 9, 2018 •

edited

Loading

ankushagarwal commented Mar 9, 2018

jimexist Mar 9, 2018

ankushagarwal Mar 9, 2018

jlewi Mar 9, 2018

jlewi left a comment

jlewi Mar 9, 2018

jlewi Mar 9, 2018

ankushagarwal Mar 9, 2018

jlewi Mar 9, 2018

jlewi Mar 9, 2018

jlewi Mar 9, 2018

ankushagarwal Mar 9, 2018

ankushagarwal Mar 9, 2018

ukclivecox Mar 10, 2018

jlewi Mar 9, 2018

ankushagarwal Mar 9, 2018

k8s-ci-robot commented Mar 9, 2018

jlewi commented Mar 9, 2018

k8s-ci-robot commented Mar 9, 2018


		Start the server using `python server.py --port=8888`.
		> The model is written in Keras and when exported as a TensorFlow model seems to be incompatible with TensorFlow Serving. So we're using seldon-core to serve this model since seldon-core allows you to serve any arbitrary model. More details [here](https://github.com/kubeflow/examples/issues/11#issuecomment-371005885).


		## Sample request
		Ensure that you have the following files from the [training](training_the_model.md) step in your `notebooks` directory:

Move from a custom tornado server to a seldon-core server for serving the model #36

Move from a custom tornado server to a seldon-core server for serving the model #36

Conversation

ankushagarwal commented Mar 9, 2018 • edited Loading

ankushagarwal commented Mar 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlewi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 9, 2018

jlewi commented Mar 9, 2018

k8s-ci-robot commented Mar 9, 2018

ankushagarwal commented Mar 9, 2018 •

edited

Loading