09-model-deployment-fastapi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5939e18e-c8d0-4ac1-b4bf-a101c4bcf06f",
   "metadata": {},
   "source": [
    "# Machine Learning Deployment using FastAPI\n",
    "\n",
    "In the previous section, we've learned how to quickly deploy machine learning models with an interactive interface using Gradio. It's fast and easy to build and deploy machine learning prototype for other poeple to try. But most of the time in production, we need different kind of user interface that has more flexibility and functionality or maybe we want to integrate the model to our existin interface. In this situation, we may have to decouple our machine learning model with the interface itself. We can focus more on the deploying machine learning models and find a way to let other services communicate to our model.\n",
    "\n",
    "A web framework is a software package or library that provides a structured and standardized way to build web applications. It typically offers a collection of tools, components, and abstractions that simplify common web development tasks, such as handling HTTP requests, managing routing, interacting with databases, and generating HTML or other types of responses.\n",
    "\n",
    "![web-framework-scheme](https://ubiops.com/wp-content/uploads/2023/04/Basic-elements-of-a-data-science-web-API.png)\n",
    "\n",
    "How do services communicate with each other? They're using an Application Programming Interface. Application Programming Interfaces (or APIs for short) have been around for a long time and basically provide a standardized way of communication between two software applications that are not necessarily of the same type.\n",
    "\n",
    "Applied to our context of data science, an API allows for the communication between a web page or app and your AI application. The API opens up certain user-defined URL endpoints, which can be used to send or receive requests with data. These endpoints are not dependent on the application: if you update your algorithm, the interface will stay the same. This minimizes the work required to update the running application.\n",
    "\n",
    "You can read more details about how API works between two applications from [here](https://hygraph.com/blog/how-do-apis-work) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e515846-346c-487a-ad2d-a82d4065ac60",
   "metadata": {},
   "source": [
    "## FastAPI Web Framework\n",
    "\n",
    "FastAPI is a modern, high-performance web framework for building APIs (Application Programming Interfaces) with Python. It is **designed to be fast, easy to use, and highly efficient**, making it a popular choice for developing web applications and microservices.\n",
    "\n",
    "One of the main features of FastAPI is its ability to **generate highly efficient code** by leveraging the type annotations introduced in Python 3.6 and above. By using type hints, **FastAPI can automatically validate data, handle request and response serialization**, and generate interactive API documentation.\n",
    "\n",
    "While FastAPI is a popular choice for deploying machine learning models, it's important to note that **there are several other frameworks and tools available** for deploying ML models, and the choice ultimately **depends on your specific requirements and preferences**. However, FastAPI does offer several advantages that make it a recommended option in many cases. Here are some reasons why professionals often suggest using FastAPI for deploying machine learning models:\n",
    "\n",
    "- **Performance**: FastAPI is known for its high performance and low latency. It leverages asynchronous programming techniques and supports asynchronous libraries like TensorFlow and PyTorch, making it suitable for handling computationally intensive ML tasks efficiently. This can be crucial, especially when dealing with real-time or high-throughput applications.\n",
    "\n",
    " - **Integration with ML libraries**: FastAPI integrates seamlessly with popular Python ML libraries and frameworks such as TensorFlow, PyTorch, scikit-learn, and more. This makes it easy to incorporate your trained ML models into the API and leverage the rich functionality provided by these libraries.\n",
    "\n",
    "- **Type annotations and validation**: FastAPI utilizes Python's type annotations to automatically serialize and validate request and response data. This ensures that the input data provided to your ML models is in the expected format, reducing the chances of errors and improving the overall reliability of your API.\n",
    "\n",
    "- **Interactive documentation**: FastAPI automatically generates detailed API documentation, including information about endpoints, input/output data types, and request/response examples. This documentation is interactive and helps developers and users understand and interact with your ML API more effectively.\n",
    "\n",
    "- **Scalability and concurrency**: FastAPI is designed to handle high levels of concurrency and can efficiently serve multiple API requests simultaneously. This is beneficial when deploying ML models that require handling multiple requests concurrently, especially in scenarios with high traffic or real-time predictions.\n",
    "\n",
    "- **Deployment options**: FastAPI can be easily deployed on various platforms, including cloud services and containerization technologies like Docker. It provides flexibility in choosing the deployment option that best suits your needs, whether it's deploying on-premises or in the cloud.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14de37b9-f822-4d86-baaf-dc7e0a7b842e",
   "metadata": {},
   "source": [
    "### Sending Data over API\n",
    "\n",
    "<img src=\"https://uploads.sitepoint.com/wp-content/uploads/2022/08/1661749125REST-API-Request.png\" width=600>\n",
    "\n",
    "When sending data over HTTP, there are several ways to include the data in the request:\n",
    "\n",
    "- **Query Parameters**: Data can be sent as part of the URL query parameters. This method is commonly used in GET requests, where the data is appended to the URL in a key-value format. Query parameters are limited in size and may not be suitable for sending large amounts of data.\n",
    "\n",
    "- **Request Body**: Data can be sent in the body of the HTTP request. This is typically done using methods like POST, PUT, and PATCH. The request body can contain various data formats, such as JSON, XML, or form data. This method allows for sending larger amounts of data and is more flexible in terms of the data structure.\n",
    "\n",
    "- **Headers**: Data can also be included in the request headers. Headers are used to provide additional information about the request, and certain headers can be used to send data, such as authentication tokens or metadata. However, headers have limitations on the amount of data they can carry, and they may not be suitable for sending large payloads.\n",
    "\n",
    "When choosing how to send data over HTTP, there are tradeoffs to consider:\n",
    "\n",
    "- **Data Size and Performance**: Sending data in query parameters may be suitable for small amounts of data but can become impractical for larger payloads. Sending large amounts of data in the request body allows for more flexibility, but it may impact performance due to increased payload size and longer transfer times.\n",
    "\n",
    "- **Security**: Depending on the nature of the data, its sensitivity, and the communication requirements, different methods may have varying security implications. For example, sending sensitive data as part of the URL query parameters may expose it in server logs or browser history, while sending it in the request body allows for better confidentiality.\n",
    "\n",
    "- **Encoding and Parsing**: Different data formats require proper encoding and parsing on both the client and server sides. Ensure compatibility between the client and server in terms of data format and handling to avoid issues with data integrity or interpretation.\n",
    "\n",
    "- **Server Constraints and API Design**: Consider the capabilities and constraints of the server and any limitations imposed by the API design. For example, some servers may have restrictions on the size of the request body or the maximum length of query parameters.\n",
    "\n",
    "I'm not going to explain everything in here, but you can read the details from [here](https://www.sitepoint.com/rest-api/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93ecf0c8-4809-45d3-ac4d-77345c012d7f",
   "metadata": {},
   "source": [
    "## Before Running\n",
    "\n",
    "- Because we can't run FastAPI inside a Notebook, we have to use linux terminal to run the python script.\n",
    "- Instead of using `torch` and `torchvision`, I'll introduce you with `onnxruntime` to execute onnx model and `albumentations` to process the augmentation pipeline\n",
    "- We'll use the Notebook to explain the process and send a request to our app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fcc12a23-f2c9-4083-96f1-dcf24835f568",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[33mWARNING: You are using pip version 21.2.4; however, version 23.1.2 is available.\n",
      "You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install requirements\n",
    "! pip -q install onnxruntime albumentations scipy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e5fa14a-f673-461f-a7fc-02eef3a1e44c",
   "metadata": {},
   "source": [
    "## Launching Simple App\n",
    "\n",
    "Similar to the previous section, we're going to deploy a simple image classfier. Before we go building a more complex system, we're going to start by creating an endpoint with these speifications:\n",
    "- expects an image encoded as a base64 string in the request body\n",
    "- return a dictionary of scores as JSON response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3648f2db-c565-4948-a79a-a5ede9cbc262",
   "metadata": {},
   "source": [
    "### Dummy App Code Sample\n",
    "\n",
    "```python\n",
    "from fastapi import FastAPI\n",
    "from pydantic import BaseModel\n",
    "import base64\n",
    "\n",
    "# create an fastapi instance\n",
    "app = FastAPI()\n",
    "\n",
    "# create input data model/definition\n",
    "# useful to specify and validate incoming input\n",
    "class ImageInput(BaseModel):\n",
    "    image_base64: str\n",
    "\n",
    "@app.post(\"/classify\")\n",
    "def classify_image(image: ImageInput):\n",
    "    # Decode the base64 image string\n",
    "    image_data = base64.b64decode(image.image_base64)\n",
    "\n",
    "    # Process the image (dummy code)\n",
    "    # Replace this with your actual machine learning model prediction code\n",
    "    # Here, we assume the image is classified as 80% cat and 20% dog\n",
    "    classification_results = {\"cat\": 0.8, \"dog\": 0.2}\n",
    "\n",
    "    return classification_results\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf38fac-0578-48d1-8c8f-d82dbd5b3a9f",
   "metadata": {},
   "source": [
    "- **Importing Dependencies**: The necessary dependencies are imported, including `FastAPI` for creating the API, `BaseModel` from `pydantic` for defining the input data model, and `base64` for decoding the base64 image string.\n",
    "\n",
    "- **Creating the FastAPI Application**: An instance of `FastAPI` is created, named app, which represents the web application.\n",
    "\n",
    "- **Defining the Input Data Model**: The ImageInput class is defined as a `BaseModel` subclass. It contains a single attribute image_base64 of type str. This class is used to specify the structure and validation rules for the input data expected by the `classify_image` endpoint.\n",
    "\n",
    "- **Defining the Classification Endpoint**: The `@app.post(\"/classify\")` decorator is used to define a POST endpoint at the `/classify` URL path. When a POST request is made to this endpoint, the classify_image function is invoked.\n",
    "\n",
    "- **Processing the Image**: Within the `classify_image` function, the base64-encoded image string is decoded using `base64.b64decode`. This converts the string back into its binary image data representation.\n",
    "\n",
    "- **Machine Learning Model (Dummy Code)**: Next, the code includes a placeholder for the machine learning model prediction code. In the provided dummy code, a dictionary named `classification_results` is created, assuming the image is classified as 80% cat and 20% dog. This section should be replaced with the actual machine learning model prediction code.\n",
    "\n",
    "- **Returning Classification Results**: The `classification_results` dictionary is returned as the response from the API endpoint. This data will be serialized into JSON format automatically by `FastAPI`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89151dda-6ba6-4e08-861c-824e204a9195",
   "metadata": {},
   "source": [
    "### Execute app\n",
    "Open a new terminal session in the notebook and run the following command to start the server:\n",
    "\n",
    "```bash\n",
    "uvicorn fastapi_dummy_app:app\n",
    "```\n",
    "\n",
    "After running the command, don't close the terminal session and return back to this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3f1b13-0288-4e0c-a0c8-03c648830e09",
   "metadata": {},
   "source": [
    "### Test Endpoint\n",
    "\n",
    "Since the `app` is running we can try to communicate with it using `POST` method. This script reads an image file, encodes it into base64, creates a payload with the base64-encoded image, sends a POST request to the `/classify` endpoint, and prints the classification results if the response is successful (status code 200) or prints the error message if an error occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5780576d-5119-4682-aaf8-0ae761dd87d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Results:  {'cat': 0.8, 'dog': 0.2}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import base64\n",
    "\n",
    "# Encode the image file to base64\n",
    "def encode_image_to_base64(image_path):\n",
    "    with open(image_path, \"rb\") as image_file:\n",
    "        encoded_string = base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
    "    return encoded_string\n",
    "\n",
    "# Image file path\n",
    "image_path = \"samples/cannoli.jpg\"\n",
    "\n",
    "# Encode the image file to base64\n",
    "image_base64 = encode_image_to_base64(image_path)\n",
    "\n",
    "# API endpoint URL\n",
    "url = \"http://localhost:8000/classify\"  # Update with the correct URL\n",
    "\n",
    "# Payload data\n",
    "payload = {\"image_base64\": image_base64}\n",
    "\n",
    "# Send POST request\n",
    "response = requests.post(url, json=payload)\n",
    "\n",
    "# Check response status code\n",
    "if response.status_code == 200:\n",
    "    # Successful response\n",
    "    classification_results = response.json()\n",
    "    print(\"Classification Results: \", classification_results)\n",
    "else:\n",
    "    # Error occurred\n",
    "    print(\"Error:\", response.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0063fa66-8475-49a1-9740-2c50e042797a",
   "metadata": {},
   "source": [
    "## Launching Basic App\n",
    "\n",
    "Now that we're able to communicate with our application, let's integrate our machine learning model into the application and build a new `basic_app.py` In this step, we're want to remove `torch` dependency by using `ONNX Runtime` to process the model and use `albumentations` for performing augmentation instead of `torchvision`. This could be beneficial if we want to reduce the application size by using `numpy` library that is a lot smaller than `torch` library size."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "899dd016-b74e-40f3-bd1d-7e2a0052fbb8",
   "metadata": {},
   "source": [
    "### Convert ONNX Model\n",
    "\n",
    "We need to convert the pretrained torch model into onnx model format. This is important to reduce the latency by optimizing the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e9cb28ed-4320-4abe-b4f1-04175d34ab08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== Diagnostic Run torch.onnx.export version 2.1.0a0+fe05266 ===========\n",
      "verbose: False, log level: Level.ERROR\n",
      "======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import torch\n",
    "from src.models import ResNet18, BasicBlock\n",
    "\n",
    "PRETRAINED_MODEL = 'pretrained/simple-lightning-epoch100/resnet18_epoch99.ckpt'\n",
    "\n",
    "model = ResNet18(3, 10)\n",
    "\n",
    "checkpoint = torch.load(PRETRAINED_MODEL)\n",
    "\n",
    "# The state dict will contains net.layer_name\n",
    "# Our model doesn't contains `net.` so we have to rename it\n",
    "state_dict = checkpoint['state_dict']\n",
    "for key in list(state_dict.keys()):\n",
    "    if 'net.' in key:\n",
    "        state_dict[key.replace('net.', '')] = state_dict[key]\n",
    "        del state_dict[key]\n",
    "\n",
    "model.load_state_dict(state_dict)\n",
    "model.eval()\n",
    "\n",
    "input = torch.rand(1, 3, 384, 384)\n",
    "_ = model(input)\n",
    "\n",
    "# Export the model\n",
    "torch.onnx.export(model,                     # model being run\n",
    "                  input,                     # model input (or a tuple for multiple inputs)\n",
    "                  \"food101_resenet18.onnx\",  # where to save the model (can be a file or file-like object)\n",
    "                  export_params=True,        # store the trained parameter weights inside the model file\n",
    "                  do_constant_folding=True,  # whether to execute constant folding for optimization\n",
    "                  input_names = ['input'],   # the model's input names\n",
    "                  output_names = ['output'], # the model's output names\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6876e1ba-016c-49df-baec-654b9c3679ac",
   "metadata": {},
   "source": [
    "### Basic App Code Sample\n",
    "\n",
    "Here is the sample code to our basic application. It is similar from the previous dummy application, but we use a model processing pipeline to produce real result instead of a dummy process and dummy result.\n",
    "\n",
    "```python\n",
    "import base64\n",
    "\n",
    "import albumentations as A\n",
    "import cv2\n",
    "import numpy as np\n",
    "import onnxruntime as ort\n",
    "from fastapi import FastAPI\n",
    "from pydantic import BaseModel\n",
    "from scipy.special import softmax\n",
    "\n",
    "# Dataset Metadata\n",
    "RGB_MEAN = [0.51442681, 0.43435301, 0.33421855]\n",
    "RGB_STD = [0.24099932, 0.246478, 0.23652802]\n",
    "\n",
    "# Transformation pipeline using Albumentations\n",
    "transformation_pipeline = A.Compose([\n",
    "    A.CenterCrop(width=384, height=384),\n",
    "    A.Normalize(mean=RGB_MEAN, std=RGB_STD)\n",
    "])\n",
    "\n",
    "# Load the ONNX model to onnxruntime\n",
    "onnx_model_path = 'food101_resenet18.onnx'\n",
    "model = ort.InferenceSession(onnx_model_path)  # Update with the correct model path\n",
    "\n",
    "# Get model input/output names\n",
    "input_name = model.get_inputs()[0].name\n",
    "output_name = model.get_outputs()[0].name\n",
    "\n",
    "class_names = ['apple_pie', 'bibimbap', 'cannoli', 'edamame', 'falafel', 'french_toast', 'ice_cream', 'ramen', 'sushi', 'tiramisu']\n",
    "class_names.sort()\n",
    "\n",
    "app = FastAPI()\n",
    "\n",
    "class ImageInput(BaseModel):\n",
    "    image_base64: str\n",
    "\n",
    "def preprocess_image(image: np.ndarray):\n",
    "    \"\"\"Preprocess the input image.\n",
    "\n",
    "    Note that the input image is in RGB mode.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    image: np.ndarray\n",
    "        Input image from callback.\n",
    "    \"\"\"\n",
    "\n",
    "    image = transformation_pipeline(image=image)['image']\n",
    "    image = np.transpose(image, (2, 1, 0))\n",
    "    image = np.expand_dims(image, axis=0)\n",
    "\n",
    "    return image\n",
    "\n",
    "@app.post(\"/classify\")\n",
    "def classify_image(image_input: ImageInput):\n",
    "    # Decode the base64 image string\n",
    "    image_data =  np.fromstring(base64.b64decode(image_input.image_base64), np.uint8)\n",
    "    image = cv2.imdecode(image_data, cv2.IMREAD_COLOR)\n",
    "\n",
    "    # If input not valid, return dummy data or raise error\n",
    "    if image is None:\n",
    "        return {\"cat\": 0.8, \"dog\": 0.2}\n",
    "\n",
    "    # Preprocess image\n",
    "    processed_image = preprocess_image(image)\n",
    "\n",
    "    # Run inference using the ONNX model\n",
    "    prediction = model.run([output_name], {input_name: processed_image})[0] # takes the first output\n",
    "\n",
    "    # Postprocess result\n",
    "    prediction = softmax(prediction, axis=1)[0] # Apply softmax to normalize the output\n",
    "    labeled_result = {name:score for name, score in zip(class_names, prediction.tolist())}\n",
    "\n",
    "    return labeled_result\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aa01712-eeb6-4a55-ada2-18a0a132fcc1",
   "metadata": {},
   "source": [
    "- **Importing Dependencies**: The necessary dependencies are imported, including `base64` for base64 decoding, `albumentations` for image transformations, `cv2` for image decoding, `numpy` for array operations, `onnxruntime` for running the ONNX model, and `FastAPI` for creating the API.\n",
    "\n",
    "- **Dataset Metadata and Transformation Pipeline**: The code defines RGB mean and standard deviation values for dataset normalization. It also creates a transformation pipeline using `Albumentations`, which performs a center crop and normalizes the image using the defined mean and standard deviation.\n",
    "\n",
    "- **Loading the ONNX Model**: The ONNX model is loaded using ONNX Runtime (`ort.InferenceSession`). The path to the model file is provided as a parameter.\n",
    "\n",
    "- **Preprocessing Function**: The `preprocess_image` function takes an input image and applies the defined transformation pipeline. The image is transposed, expanded, and returned as a preprocessed image.\n",
    "\n",
    "- **/classify Endpoint**: The `/classify endpoint` is defined using the `@app.post decorator`. When a POST request is made to this endpoint, the `classify_image` function is invoked.\n",
    "\n",
    "- **Image Decoding and Preprocessing**: The `classify_image` function decodes the base64 image string using `base64.b64decode` and `cv2.imdecode`. It checks if the image is valid and if not, returns dummy data. If the image is valid, it preprocesses the image using the preprocess_image function.\n",
    "\n",
    "- **Running Inference**: The preprocessed image is passed to the ONNX model using `model.run`. The output predictions are retrieved and stored in the prediction variable.\n",
    "\n",
    "- **Postprocessing**: The prediction scores are postprocessed by applying the softmax function to normalize the output probabilities. The class names and corresponding scores are combined into a dictionary, labeled_result, where the class name is the key and the score is the value.\n",
    "\n",
    "- **Returning the Result**: The labeled_result dictionary is returned as the response from the `/classify endpoint`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6fbe898-5504-45b9-ad35-1ff304003f43",
   "metadata": {},
   "source": [
    "### Execute app\n",
    "Close the previous terminal seesion by pressing `CTRL+C` or close the terminal window. Atter that, run the following command to start the server:\n",
    "\n",
    "```\n",
    "uvicorn fastapi_basic_app:app\n",
    "```\n",
    "\n",
    "After running the command, don't close the terminal session and return back to this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5cf4777-1707-4f7c-a3e9-96b4bcae225c",
   "metadata": {},
   "source": [
    "### Test Endpoint\n",
    "\n",
    "Since the `app` is running we can try to communicate with it using `POST` method. This script reads an image file, encodes it into base64, creates a payload with the base64-encoded image, sends a POST request to the `/classify` endpoint, and prints the classification results if the response is successful (status code 200) or prints the error message if an error occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b91a0493-74d3-48a7-8a14-181fb7519366",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Results:  {'apple_pie': 0.007718801964074373, 'bibimbap': 2.0401820677307114e-07, 'cannoli': 0.9777591228485107, 'edamame': 1.0776185263239313e-05, 'falafel': 0.0001419971522409469, 'french_toast': 0.014265178702771664, 'ice_cream': 0.00010071938595501706, 'ramen': 1.887471157147047e-08, 'sushi': 2.448658960929606e-06, 'tiramisu': 6.506219847324246e-07}\n"
     ]
    }
   ],
   "source": [
    "# Image file path\n",
    "image_path = \"samples/cannoli.jpg\"\n",
    "\n",
    "# Encode the image file to base64\n",
    "image_base64 = encode_image_to_base64(image_path)\n",
    "\n",
    "# API endpoint URL\n",
    "url = \"http://localhost:8000/classify\"  # Update with the correct URL\n",
    "\n",
    "# Payload data\n",
    "payload = {\"image_base64\": image_base64}\n",
    "\n",
    "# Send POST request\n",
    "response = requests.post(url, json=payload)\n",
    "\n",
    "# Check response status code\n",
    "if response.status_code == 200:\n",
    "    # Successful response\n",
    "    classification_results = response.json()\n",
    "    print(\"Classification Results: \", classification_results)\n",
    "else:\n",
    "    # Error occurred\n",
    "    print(\"Error:\", response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267b4f0b-88b0-410d-9c57-b899ed3f8603",
   "metadata": {},
   "source": [
    "This example should be enough to give you on how to deploy the application using a web framework. Once the application is ready, you can deploy the application on a cloud or vm according to your needs. The best way to deploy your model is to use a Docker, you can read more about it [here](https://docker-curriculum.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1cc10ef-253d-4409-898e-5fd863aecd73",
   "metadata": {},
   "source": [
    "## Scalability\n",
    "\n",
    "*The DevOps team will take care of the scaling plan, but it's important for you to know about the process because the applications contain AI models that need a significant amount of computational power.*\n",
    "\n",
    "I will explain to you about the idea of scaling a machine learning application. We can deploy our application in the cloud and let the other services communicate to our application with an API. Scalability is crucial for production because **your application must be able to handle a lot of users**. What if there there are times when the application needs to handle a huge number of users and then goes back to normal or only has a few active users? You'll have to come up with the best approach to adjust the number of applications dynamically based on user demand.\n",
    "\n",
    "The most popular tool to scale the system is using [Kubertenes](https://kubernetes.io/). Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications. This too will help deploy and manage containerized applications based on the configuration from DevOps.\n",
    "\n",
    "Let's define a few conditions for the next explanation:\n",
    "- The FastAPI **server will process a classification system using a single model** regardless of its optimization method\n",
    "- The term `user` is the other service that tries to use our server, such as backend service or a real user at some endpoint.\n",
    "\n",
    "### Single Instance Server\n",
    "\n",
    "Based on the picture below, we have a simple situation where the application deals with just one request from a user. In this case, the system runs smoothly and functions perfectly, as we have already tested and measured its performance. However, it's important to note that such instances where the application handles only one request at a time are quite uncommon. In reality, many users will attempt to use the application simultaneously, which can pose challenges. This particular scenario of handling a single request is more likely to occur during the testing phase of the application.\n",
    "\n",
    "<center><img alt=\"single-user-single-server\" src=\"figures/single-user-single-instance.png\" width=600></center>\n",
    "\n",
    "Now let's look at the next picture, where we have a maximum of 60 users using our application. Based on the performance test results we conducted earlier, our model can handle up to 60 frames per second (FPS) or 60 requests per second (QPS). When there are exactly 60 users trying to use our application, this could be the point where additional users might start experiencing delays.\n",
    "\n",
    "When there are 60 requests, the application can handle them in batches or process them one by one while still delivering a fast response to the user. This ensures efficient operation and quick turnaround times. However, if there are more than 60 requests occurring simultaneously, some users may experience a brief wait as the application completes processing the initial batch of 60 requests before moving on to the next batch, and so on.\n",
    "\n",
    "Essentially, with 60 users, the application functions optimally and provides a smooth user experience. Beyond that threshold, the application may experience increased latency as it juggles multiple requests, requiring some users to wait momentarily for their turn to be processed.\n",
    "\n",
    "<center><img alt=\"multi-user-single-server\" src=\"figures/multi-user-single-instance.png\" width=600></center>\n",
    "\n",
    "### Multiple Instance Server\n",
    "\n",
    "*Don't worry if you don't understand about a load balancer, it is basiclly a routing system to ensure the traffic is distributed between the services.*\n",
    "\n",
    "When you have a large number of users, a single instance of the application may struggle to handle them all. To address this issue, we need to increase the number of application instances until our application can effectively accommodate all the users. However, it's important to keep in mind that each application instance contains a single model. Therefore, having multiple application instances will result in multiple models as well. This can potentially create a new problem where our hardware may not have sufficient memory to store all these models. As a result, the number of applications that can be created is limited by the capacity of the hardware.\n",
    "\n",
    "Additionally, there is another challenge when the number of users decreases but the number of application instances remains the same. In such cases, the applications may not be fully utilized to process the reduced number of requests efficiently. Therefore, we need to find the optimal strategy and determine the appropriate number of applications that can effectively run within the hardware constraints. This involves striking a balance between resource utilization, memory capacity, and the demands of the user base.\n",
    "\n",
    "<center><img alt=\"multi-user-multi-server\" src=\"figures/multi-user-multi-instance.png\" width=600></center>\n",
    "\n",
    "One of the best library to serve machine learning model is [Ray Serve](https://docs.ray.io/en/latest/serve/index.html). **Ray Serve** a scalable model serving library for building online inference APIs. Serve is framework-agnostic, so you can use a single toolkit to serve everything from deep learning models built with frameworks like PyTorch, Tensorflow, and Keras, to Scikit-Learn models, to arbitrary Python business logic. It is particularly well suited for model composition, enabling you to build a complex inference service consisting of multiple ML models and business logic all in Python code. Ray Serve can be used with FastAPI with its full range of features. \n",
    "\n",
    "<center><img alt=\"ray-serve-architecture\" src=\"https://docs.ray.io/en/latest/_images/architecture-2.0.svg\" width=600></center>\n",
    "\n",
    "What if we consider separating the FastAPI pipeline from the model itself? By decoupling these components, we can adjust each part independently, giving us more control over our application. However, it's important to note that this separation may introduce some additional time for data transfer between FastAPI and the model service.\n",
    "\n",
    "Now, let's imagine that we further optimize the model to handle up to 90 frames per second (FPS). In this case, a single model can handle the workload of up to 2 application servers. By implementing this strategy, we can reduce the number of models required on a single hardware setup and effectively utilize all the available models for our applications.\n",
    "\n",
    "But what if we require different types of models? In such scenarios, we can carefully select and combine the models needed on the server by adjusting the duplicates based on the demands of our application. This approach allows us to optimize the resources and achieve the best performance for our specific requirements.\n",
    "\n",
    "Since each model has its own dedicated server, we can make adjustments to the application server without worrying about GPU limitations, as the model servers handle the heavy computation tasks.\n",
    "\n",
    "<center><img alt=\"multi-user-splitted-server\" src=\"figures/multi-user-multi-instance-splitted.png\" width=600></center>\n",
    "\n",
    "One of the best tool to serve the models is [Triton Inference Server](https://github.com/triton-inference-server/server). **Triton Inference Server** or TRIS is an open source inference serving software that streamlines AI inferencing. Triton enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton supports inference across cloud, data center,edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton delivers optimized performance for many query types, including real time, batched, ensembles and audio/video streaming.\n",
    "\n",
    "<center><img alt=\"triton-inference-server\" src=\"https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/_images/triton_on_jetson.png\" width=800></center"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "490d83ef-19f8-4890-963d-e7de1d626819",
   "metadata": {},
   "source": [
    "Each method has advantages and disadvantages. You have to wisely choose which one is best for your use case."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}