t81_558_class_13_02_cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 13: Advanced/Other Topics**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 13 Video Material\n",
    "\n",
    "* Part 13.1: Flask and Deep Learning Web Services [[Video]](https://www.youtube.com/watch?v=H73m9XvKHug&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_01_flask.ipynb)\n",
    "* **Part 13.2: Deploying a Model to AWS**  [[Video]](https://www.youtube.com/watch?v=8ygCyvRZ074&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_02_cloud.ipynb)\n",
    "* Part 13.3: Using a Keras Deep Neural Network with a Web Application  [[Video]](https://www.youtube.com/watch?v=OBbw0e-UroI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_03_web.ipynb)\n",
    "* Part 13.4: When to Retrain Your Neural Network [[Video]](https://www.youtube.com/watch?v=K2Tjdx_1v9g&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_04_retrain.ipynb)\n",
    "* Part 13.5: AI at the Edge: Using Keras on a Mobile Device  [[Video]]() [[Notebook]](t81_558_class_13_05_edge.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 13.2: Deploying a Model to AWS\n",
    "\n",
    "Some additional material:\n",
    "\n",
    "* [Serving TensorFlow Models](https://www.tensorflow.org/tfx/guide/serving) - Using Google's own deployment server.\n",
    "* [Deploy trained Keras or TensorFlow models using Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker/) - Using AWS to deploy TensorFlow ProtoBuffer models.\n",
    "* [Google ProtoBuf](https://developers.google.com/protocol-buffers/) - The file format used to store neural networks for deployment.\n",
    "\n",
    "# Part 13.2.1: Train Model (optionally, outside of AWS)\n",
    "\n",
    "A portion of this part will need to be run from [AWS SageMaker](https://aws.amazon.com/sagemaker/). To do this you will need to upload this IPYNB (for Module 13.2) to AWS Sage Maker and open it from Jupyter.  This complete process is demonstrated in the above YouTube video.\n",
    "\n",
    "We begin by training a MPG dataset.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into validation and training sets\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',\n",
    "        restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we evaluate the RMSE.  The goal is more to show how to create a cloud API than to achieve a really low RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"RMSE Score: {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we save the weights and structure of the neural network, as was demonstrated earlier in this course.  These two files are used to generate a ProtoBuf file that is used for the actual deployment.  We store it to two separate files because we ONLY want the structure and weights.  A single MD5 file, such as model.save(...) also contains training paramaters and other features that may cause version issues when uploading to AWS.  Remember, AWS may have a different version for TensorFlow than you do locally.  Usually AWS will have an older version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = \"./dnn/\"\n",
    "\n",
    "model.save_weights(os.path.join(save_path,\"mpg_model-weights.h5\"))\n",
    "\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = model.to_json()\n",
    "with open(os.path.join(save_path,\"mpg_model.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will upload the two files generated to the **./dnn/** folder to AWS.  If you running the entire process from  AWS, then they will not need to be uploaded.\n",
    "\n",
    "We also print out the values to one car, we will copy these later when we test the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x[0].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 13.2.2: Convert Model (must use AWS SageMaker Notebook)\n",
    "\n",
    "To complete this portion you will need to be running from s Jupyter notebook on AWS SageMaker.  The following is based on an example from AWS documentation, but customized to this class example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1. Set up\n",
    "\n",
    "In the AWS Management Console, go to the Amazon SageMaker console. Choose Notebook Instances, and create a new notebook instance. Upload this notebook and set the kernel to conda_tensorflow_p36.\n",
    "\n",
    "The get_execution_role function retrieves the AWS Identity and Access Management (IAM) role you created at the time of creating your notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3, re\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2. Load the Keras model using the json and weights file\n",
    "\n",
    "The following cell loads the necessary imports from AWS.  Note that we using \"import keras\" compared to the \"import keras.tensorflow\" advised for the rest of the course.  This is advised by AWS currently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.models import model_from_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a directory called keras_model, navigate to keras_model from the Jupyter notebook home, and upload the model.json and model-weights.h5 files (using the \"Upload\" menu on the Jupyter notebook home)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir keras_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate to keras_model from the Jupyter notebook home, and upload your model.json and model-weights.h5 files (using the \"Upload\" menu on the Jupyter notebook home). Use the files that you generated in step 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls keras_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you've uploaded your model to the directory by this point.  If you saw no files at the above step, upload your files and rerun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "json_file = open('/home/ec2-user/SageMaker/keras_model/'+'mpg_model.json', 'r')\n",
    "loaded_model_json = json_file.read()\n",
    "json_file.close()\n",
    "loaded_model = model_from_json(loaded_model_json,custom_objects={\"GlorotUniform\": tf.keras.initializers.glorot_uniform})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model.load_weights('/home/ec2-user/SageMaker/keras_model/mpg_model-weights.h5')\n",
    "print(\"Loaded model from disk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3. Export the Keras model to the TensorFlow ProtoBuf format (must use AWS SageMaker Notebook)\n",
    "\n",
    "As you are probably noticing there are many ways to save a Keras neural network.  So far we've seen:\n",
    "\n",
    "* YAML File - Structure only\n",
    "* JSON File - Structure only\n",
    "* H5 Complete Model\n",
    "* H5 Weights only\n",
    "\n",
    "There is actually a fifth, which is the ProtoBuf format.  ProtoBuf is typically only used for deployment.  We will now convert the model we just loaded into this format.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.saved_model import builder\n",
    "from tensorflow.python.saved_model.signature_def_utils import predict_signature_def\n",
    "from tensorflow.python.saved_model import tag_constants\n",
    "\n",
    "# Note: This directory structure will need to be followed - see notes for the next section\n",
    "model_version = '1'\n",
    "export_dir = 'export/Servo/' + model_version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is very important that this export directory be empty.  Be careful, the following command deletes the entire expor directory. (this should be fine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(export_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the Protocol Buffer SavedModel at 'export_dir'\n",
    "build = builder.SavedModelBuilder(export_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create prediction signature to be used by TensorFlow Serving Predict API\n",
    "signature = predict_signature_def(\n",
    "    inputs={\"inputs\": loaded_model.input}, outputs={\"score\": loaded_model.output})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "\n",
    "with K.get_session() as sess:\n",
    "    # Save the meta graph and variables\n",
    "    build.add_meta_graph_and_variables(\n",
    "        sess=sess, tags=[tag_constants.SERVING], signature_def_map={\"serving_default\": signature})\n",
    "    build.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notive the **signature_def_map** this bridges any incompatabilities between the version of TensorFlow you were running locally and the AWS version.  You might need to add additional entries here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4. Convert TensorFlow model to a SageMaker readable format (must use AWS SageMaker Notebook)\n",
    "\n",
    "Move the TensorFlow exported model into a directory export\\Servo. SageMaker will recognize this as a loadable TensorFlow model. Your directory and file structure should look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls export/Servo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls export/Servo/1/variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Tar the entire directory and upload to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tarfile\n",
    "with tarfile.open('model.tar.gz', mode='w:gz') as archive:\n",
    "    archive.add('export', recursive=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload TAR file to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "inputs = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5. Deploy the trained model (must use AWS SageMaker Notebook)\n",
    "\n",
    "The entry_point file \"train.py\" can be an empty Python file. The requirement will be removed at a later date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!touch train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "sagemaker_model = TensorFlowModel(model_data = 's3://' + sagemaker_session.default_bucket() + '/model/model.tar.gz',\n",
    "                                  role = role,\n",
    "                                  framework_version = '1.12',\n",
    "                                  entry_point = 'train.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the following command cake take 5-8 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "predictor = sagemaker_model.deploy(initial_instance_count=1,\n",
    "                                   instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: You will need to update the endpoint in the command below with the endpoint name from the output of the previous cell (e.g. sagemaker-tensorflow-2019-07-24-01-47-19-895)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = 'sagemaker-tensorflow-2019-08-05-03-29-25-591'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "predictor=sagemaker.tensorflow.model.TensorFlowPredictor(endpoint_name, sagemaker_session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 13.2.3: Test Model Deployment (optionally, outside of AWS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import boto3\n",
    "import numpy as np\n",
    "import io\n",
    "\n",
    "endpoint_name = 'sagemaker-tensorflow-2019-08-05-03-29-25-591' # see above, must be set to current value\n",
    "\n",
    "# Pick one of the following two cells to run based on how you will access..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important, do not run both of the cells below!! Read comments**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you access the API from outside of AWS SageMaker notebooks you must authenticate and specify region...\n",
    "# (do not run both this cell and the next)\n",
    "\n",
    "client = boto3.client('runtime.sagemaker', \n",
    "    region_name='us-east-1', # make sure to set correct region\n",
    "    aws_access_key_id='AKIAYKSSG3L5P2H5EU77', # These you get from AWS, for your account\n",
    "    aws_secret_access_key='1GYDRaE1o/nFfW2nF6jAJpWrd2R5Eut/d6fS6ruL')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you access from inside AWS in a notebook (do not run both this cell and the previous)\n",
    "client = boto3.client('runtime.sagemaker', region_name='us-east-1') # make sure to set correct region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Call the end point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'{\\n  \"outputs\": {\\n    \"score\": {\\n      \"dtype\": \"DT_FLOAT\", \\n      \"floatVal\": [\\n        13.595364570617676\\n      ], \\n      \"tensorShape\": {\\n        \"dim\": [\\n          {\\n            \"size\": \"1\"\\n          }, \\n          {\\n            \"size\": \"1\"\\n          }\\n        ]\\n      }\\n    }\\n  }, \\n  \"modelSpec\": {\\n    \"version\": \"1\", \\n    \"name\": \"generic_model\", \\n    \"signatureName\": \"serving_default\"\\n  }\\n}'\n"
     ]
    }
   ],
   "source": [
    "# Create a car based on one of the cars captured at beginning of this part.\n",
    "data = [[8,307,130,3504,12,70,1]]\n",
    "\n",
    "response = client.invoke_endpoint(EndpointName=endpoint_name, Body=json.dumps(data))\n",
    "response_body = response['Body']\n",
    "print(response_body.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}