Skip to content

Commit

Permalink
update cifar10 demo (#247)
Browse files Browse the repository at this point in the history
  • Loading branch information
@ukclivecox
ukclivecox authored May 27, 2022
1 parent c9a47e0 commit f11566a
Show file tree
Hide file tree
Showing 2 changed files with 156 additions and 18 deletions.
Binary file added samples/examples/image_classifier/demo.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
174 changes: 156 additions & 18 deletions samples/examples/image_classifier/infer.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,91 @@
"source": [
"## CIFAR10 Image Classification Production Deployment\n",
"\n",
"![cifar10](demo.png)\n",
"\n",
"We show an image classifier (CIFAR10) with associated outlier and drift detectors using a Pipeline.\n",
"\n",
" * The model is a tensorflow [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) image classfier \n",
" * The outlier detector is created from the [CIFAR10 VAE Outlier example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/od_vae_cifar10.html).\n",
" * The drift detector is created from the [CIFAR10 KS Drift example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_ks_cifar10.html)\n",
" \n",
"### Model Training (optional for notebook)\n",
"\n",
"Here we describe how we created the artifacts. This is not needed to run this notebook but will help you to apply these techniques to your own models.\n",
"\n",
"#### Outlier Detection Model\n",
"\n",
"Follow the Alibi-Detect outlier detector example [CIFAR10 VAE Outlier example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/od_vae_cifar10.html).\n",
"\n",
"Use the `save_detector` command to save the trained outlier detector as shown in the notebook.\n",
"\n",
"Create a MLServer model settings file: `model-settings.json`:\n",
"\n",
"```json\n",
"{\n",
" \"name\": \"cifar10-outlier-detect\",\n",
" \"implementation\": \"mlserver_alibi_detect.AlibiDetectRuntime\",\n",
" \"parameters\": {\n",
" \"uri\": \"./\",\n",
" \"version\": \"v0.1.0\"\n",
" }\n",
"}\n",
"\n",
"```\n",
"\n",
"Save to local or remote storage the directory. Here we saved to Google Storage:\n",
"\n",
"```bash\n",
"gsutil ls -R gs://seldon-models/mlserver/alibi-detect/cifar10-outlier \n",
"\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/:\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/OutlierVAE.dill\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/meta.dill\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model-settings.json\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/:\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/checkpoint\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/decoder_net.h5\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/encoder_net.h5\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/vae.ckpt.data-00000-of-00001\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-outlier/model/vae.ckpt.index\n",
"\n",
"```\n",
" \n",
" ### Drift Detector\n",
" \n",
"Follow the Alibi-Detect drift detection example [CIFAR10 KS Drift example](https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_ks_cifar10.html).\n",
"\n",
"Use the `save_detector` command to save the trained outlier detector as shown in the notebook.\n",
"\n",
"Create a MLServer model settings file: `model-settings.json`:\n",
"\n",
"```json\n",
"{\n",
" \"name\": \"cifar10-drift\",\n",
" \"implementation\": \"mlserver_alibi_detect.AlibiDetectRuntime\",\n",
" \"parameters\": {\n",
" \"uri\": \"./\",\n",
" \"version\": \"v0.1.0\"\n",
" }\n",
"}\n",
"```\n",
"\n",
"Save to local or remote storage the directory. Here we saved to Google Storage:\n",
"\n",
"```bash\n",
"gsutil ls -R gs://seldon-models/mlserver/alibi-detect/cifar10-drift \n",
"\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/:\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/KSDrift.dill\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/meta.dill\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/model-settings.json\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/model/:\n",
"gs://seldon-models/mlserver/alibi-detect/cifar10-drift/model/encoder.h5\n",
"\n",
"```\n",
" \n",
" "
]
},
Expand All @@ -25,8 +105,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2022-05-26 11:27:37.239516: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
"2022-05-26 11:27:37.239536: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
"2022-05-27 08:41:56.822108: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
"2022-05-27 08:41:56.822126: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
]
}
],
Expand Down Expand Up @@ -154,7 +234,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 36,
"id": "close-people",
"metadata": {},
"outputs": [
Expand Down Expand Up @@ -206,7 +286,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 37,
"id": "assumed-bolivia",
"metadata": {},
"outputs": [
Expand All @@ -228,7 +308,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 38,
"id": "specified-observer",
"metadata": {},
"outputs": [
Expand All @@ -250,7 +330,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 39,
"id": "interior-tokyo",
"metadata": {},
"outputs": [
Expand Down Expand Up @@ -283,7 +363,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 40,
"id": "modified-weekly",
"metadata": {},
"outputs": [
Expand All @@ -301,7 +381,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 41,
"id": "mechanical-receptor",
"metadata": {},
"outputs": [
Expand All @@ -315,7 +395,7 @@
" {\r\n",
" \"pipeline\": {\r\n",
" \"name\": \"cifar10-production\",\r\n",
" \"uid\": \"ca7lvtv7a4umaeddt0gg\",\r\n",
" \"uid\": \"ca88r1j2ekr7o49ljqkg\",\r\n",
" \"version\": 1,\r\n",
" \"steps\": [\r\n",
" {\r\n",
Expand Down Expand Up @@ -345,7 +425,7 @@
" \"pipelineVersion\": 1,\r\n",
" \"status\": \"PipelineReady\",\r\n",
" \"reason\": \"Created pipeline\",\r\n",
" \"lastChangeTimestamp\": \"2022-05-26T11:09:44.988745665Z\"\r\n",
" \"lastChangeTimestamp\": \"2022-05-27T08:36:23.489134152Z\"\r\n",
" }\r\n",
" }\r\n",
" ]\r\n",
Expand All @@ -359,7 +439,7 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 42,
"id": "featured-abortion",
"metadata": {},
"outputs": [
Expand All @@ -378,17 +458,17 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 43,
"id": "herbal-scratch",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"---\r\n",
"seldon.default.model.cifar10-drift.outputs\r\n",
"{\"name\":\"is_drift\", \"datatype\":\"INT64\", \"shape\":[\"1\"], \"contents\":{\"int64Contents\":[\"1\"]}}\r\n"
"---\n",
"seldon.default.model.cifar10-drift.outputs\n",
"{\"name\":\"is_drift\", \"datatype\":\"INT64\", \"shape\":[\"1\"], \"contents\":{\"int64Contents\":[\"1\"]}}\n"
]
}
],
Expand All @@ -398,7 +478,65 @@
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 44,
"id": "1ec8c10c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<Response [200]>\n",
"{'model_name': '', 'outputs': [{'data': None, 'name': 'fc10', 'shape': [1, 10], 'datatype': 'FP32'}, {'data': [1], 'name': 'is_outlier', 'shape': [1, 1], 'datatype': 'INT64'}], 'rawOutputContents': ['uGMYNZnyHTje30c1ozCjOyFeLTSlAMg2iLZ+P3hsiDOGenI11U+dNQ==']}\n"
]
}
],
"source": [
"infer(\"cifar10-production.pipeline\",1, \"outlier\")"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "5fdefbc4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<Response [200]>\n",
"{'model_name': '', 'outputs': [{'data': None, 'name': 'fc10', 'shape': [1, 10], 'datatype': 'FP32'}, {'data': [0], 'name': 'is_outlier', 'shape': [1, 1], 'datatype': 'INT64'}], 'rawOutputContents': ['JRx5MgYnrDCDAC802B7sPd4qOzQ1js82tXtiP3my0DFHup8zpZSiMQ==']}\n"
]
}
],
"source": [
"infer(\"cifar10-production.pipeline\",1, \"ok\")"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "6e88dfda",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"---\r\n",
"seldon.default.model.cifar10.outputs\r\n",
"{\"modelName\":\"cifar10_1\", \"modelVersion\":\"1\", \"outputs\":[{\"name\":\"fc10\", \"datatype\":\"FP32\", \"shape\":[\"1\", \"10\"], \"contents\":{\"fp32Contents\":[1.4500107e-8, 1.2525738e-9, 1.6298331e-7, 0.115293205, 1.7431327e-7, 0.000006185636, 0.8847001, 6.0738867e-9, 7.437898e-8, 4.7317195e-9]}}], \"rawOutputContents\":[\"JRx5MgYnrDCDAC802B7sPd4qOzQ1js82tXtiP3my0DFHup8zpZSiMQ==\"]}\r\n"
]
}
],
"source": [
"!seldon pipeline inspect cifar10-production.cifar10.outputs"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "increased-zealand",
"metadata": {},
"outputs": [
Expand All @@ -411,12 +549,12 @@
}
],
"source": [
"!seldon pipeline unload -p cifar10-production"
"!seldon pipeline unload cifar10-production"
]
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 35,
"id": "handled-continuity",
"metadata": {},
"outputs": [
Expand Down

0 comments on commit f11566a

Please sign in to comment.