covid_diagnostic.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "covid_diagnostic.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "file_extension": ".py",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "mimetype": "text/x-python",
    "name": "python",
    "npconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": 3
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WSMO2e0Ca3tU"
      },
      "source": [
        "## **Covid Diagnosis Using Chest X-Ray Data**\n",
        "***Important***: Before starting this question, please copy the folder into your own Google Drive, which you can do by downloading and then uploading to your Drive, or by doing `File > Save a Copy in Drive`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n_Kecj9IbFCT"
      },
      "source": [
        "### Introduction\n",
        "COVID-19 is severely impacting the health of countless people worldwide. A crucial step in controlling the disease has been early detection of infected patients, which can be achieved through radiography, according to prior literature that shows COVID-19 causes chest abnormalities noticeable in chest X-rays. With this in mind, you will build a model that learns chest abnormalities and predicts whether a patient has COVID-19 based on his/her chest X-ray. To do this, you will utilize transfer learning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3tF64VybOiZ"
      },
      "source": [
        "We begin by importing necessary packages for this model. You will need to import more packages along the way, so feel free to add to this list or import as you go.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZKzt0tiAbYLY"
      },
      "source": [
        "import cv2\n",
        "import os\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zd6HN-95LqYr"
      },
      "source": [
        "###Data Collection\n",
        "There is no substantially-sized, clinically verified, and publicly available COVID-19 dataset. However, a small composite dataset with X-Rays of COVID-19 positive patients recently became publicly available with [DeepCovid](https://github.com/shervinmin/DeepCovid), which compiled their data from:\n",
        "\n",
        "[Covid-Chestxray-Dataset](https://github.com/ieee8023/covid-chestxray-dataset) for COVID-19 X-ray samples\n",
        "\n",
        "[ChexPert Dataset](https://stanfordmlgroup.github.io/competitions/chexpert/) for Non-COVID samples\n",
        "\n",
        "If you do not have Google Colab Pro, it is highly recommended that you invest in this resource if you haven't already; however, this is not required, as you will not use the entire data set, but instead work with a subset of the data that we have compressed for you. To access this data, which is in the midterm folder in your Drive, you must begin by connecting to your Drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4VyDyG1pJHZ9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d175bbe-bcbc-4368-d0ec-867aadb79cb7"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QAZ_MQOOBet"
      },
      "source": [
        "Now that we have connected our Colab to our Drive, we can find the path to the data.zip file and unzip it with ZipFile."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GaQiCm9YJbw0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "caa1eb2a-7caf-4bb6-a565-308b647e1d0a"
      },
      "source": [
        "from zipfile import ZipFile\n",
        "\n",
        "file_name = 'drive/My Drive/Midterm/dataset.zip'\n",
        "\n",
        "with ZipFile(file_name, 'r') as zip:\n",
        "  zip.extractall('drive/My Drive/Midterm/dataset')\n",
        "  print(\"Data extracted!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data extracted!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J49a13qRDnau",
        "outputId": "c3955bc9-3d35-4cdc-ac10-69e768d79eaa"
      },
      "source": [
        "image = cv2.imread('drive/My Drive/Midterm/dataset/dataset/covid/000001-1.jpg')\r\n",
        "image.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(768, 968, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4VmIvYDDRJR6"
      },
      "source": [
        "Next, let us begin feature and label building. We can utilize Paths to \n",
        "list the directory of every X-Ray from the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rO_jTgD5H1Zo"
      },
      "source": [
        "from imutils import paths\n",
        "\n",
        "data = []\n",
        "labels = []\n",
        "\n",
        "#TODO: Grab list of image paths using paths.list_images\n",
        "imagePaths = sorted(list(paths.list_images(\"drive/My Drive/Midterm/dataset/dataset/\")))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n3aQDlGbF0XN"
      },
      "source": [
        "#TODO: Create a list of labels and image data by looping over the image paths\r\n",
        "#Hint: You can extract the class label based on the image's filepath\r\n",
        "#NOTE: Before adding the image to your list of images, you should do some preprocessing\r\n",
        "#      Think about what the input of ResNet is; specifically, think about color \r\n",
        "#      channels and resizing (use cv2!).\r\n",
        "\r\n",
        "for imagePath in imagePaths:\r\n",
        "\t# extract the class label from the filename\r\n",
        "  label = imagePath.split(\"/\")[-2]\r\n",
        "\r\n",
        "  image = cv2.imread(imagePath)\r\n",
        "  image = cv2.resize(image, (224, 224))\r\n",
        "  image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\r\n",
        "\r\n",
        "\r\n",
        "  data.append(image)\r\n",
        "  labels.append(label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "6hhjt5ZDHSRS",
        "outputId": "a09c1fbd-fc12-4fba-a35a-21abc8a5c1c9"
      },
      "source": [
        "s = 'drive/My Drive/Midterm/dataset/dataset/covid/aspiration-pneumonia-5-day0.jpg'\r\n",
        "s.split(\"/\")[-2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'covid'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0920nm76d4Hs"
      },
      "source": [
        "Yikes, we're using a Python list to hold our data. Let's convert that to a Numpy Array and normalize the pixel values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6QNL7ibcdzIj"
      },
      "source": [
        "data = np.array(data) / 255.0\n",
        "labels = np.array(labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n3zmzxdQd9ON"
      },
      "source": [
        "Our labels still need a little help. Let's use SKLearn to one-hot encode our labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0J2yOK7md_y6"
      },
      "source": [
        "from sklearn.preprocessing import LabelBinarizer\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "lb = LabelBinarizer()\n",
        "labels = lb.fit_transform(labels)\n",
        "labels = to_categorical(labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHZEuVVDeBv9"
      },
      "source": [
        "### Data Splitting\n",
        "Now that we're done grabbing our data, we can begin to look at splitting the data into our training and validation sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FK0h4tnQeH56"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "#TODO: Partition data into 85% training and 15% validation\n",
        "\n",
        "(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.15, random_state=42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_echi_0APEis"
      },
      "source": [
        "### Set hyperparameters\n",
        "Before building the model, it's always important to set your relevant hyperparameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FweemVCFPH9c"
      },
      "source": [
        "EPOCHS = 20\n",
        "BATCH_SIZE = 16\n",
        "LR = 1e-4"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YByvq-9oRwn_"
      },
      "source": [
        "### Base model\n",
        "Let's load our ResNet model. Our pretrained weights should come from imagenet, and our input is (224,224,3) as mentioned earlier. Make sure to not include the fully-connected layers of the model as we are dealing with different output dimensions!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "InLljPNaeYI2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab76d748-8255-421f-e8fb-9a1bec63b49a"
      },
      "source": [
        "#TODO: load in Resnet\n",
        "# from tensorflow.keras.applications import VGG16\n",
        "from tensorflow.keras.applications import ResNet50\n",
        "from tensorflow.keras.layers import Input\n",
        "\n",
        "input_ = Input(shape=(224, 224, 3))\n",
        "base = ResNet50(include_top = False, \n",
        "                weights='imagenet',\n",
        "                input_tensor = input_)\n",
        "\n",
        "base.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"resnet50\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 224, 224, 3) 0                                            \n",
            "__________________________________________________________________________________________________\n",
            "conv1_pad (ZeroPadding2D)       (None, 230, 230, 3)  0           input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "conv1_conv (Conv2D)             (None, 112, 112, 64) 9472        conv1_pad[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "conv1_bn (BatchNormalization)   (None, 112, 112, 64) 256         conv1_conv[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "conv1_relu (Activation)         (None, 112, 112, 64) 0           conv1_bn[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "pool1_pad (ZeroPadding2D)       (None, 114, 114, 64) 0           conv1_relu[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "pool1_pool (MaxPooling2D)       (None, 56, 56, 64)   0           pool1_pad[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_1_conv (Conv2D)    (None, 56, 56, 64)   4160        pool1_pool[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block1_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_1_relu (Activation (None, 56, 56, 64)   0           conv2_block1_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_2_conv (Conv2D)    (None, 56, 56, 64)   36928       conv2_block1_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_2_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block1_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_2_relu (Activation (None, 56, 56, 64)   0           conv2_block1_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_0_conv (Conv2D)    (None, 56, 56, 256)  16640       pool1_pool[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_3_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block1_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_0_bn (BatchNormali (None, 56, 56, 256)  1024        conv2_block1_0_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_3_bn (BatchNormali (None, 56, 56, 256)  1024        conv2_block1_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_add (Add)          (None, 56, 56, 256)  0           conv2_block1_0_bn[0][0]          \n",
            "                                                                 conv2_block1_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block1_out (Activation)   (None, 56, 56, 256)  0           conv2_block1_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_1_conv (Conv2D)    (None, 56, 56, 64)   16448       conv2_block1_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block2_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_1_relu (Activation (None, 56, 56, 64)   0           conv2_block2_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_2_conv (Conv2D)    (None, 56, 56, 64)   36928       conv2_block2_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_2_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block2_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_2_relu (Activation (None, 56, 56, 64)   0           conv2_block2_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_3_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block2_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_3_bn (BatchNormali (None, 56, 56, 256)  1024        conv2_block2_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_add (Add)          (None, 56, 56, 256)  0           conv2_block1_out[0][0]           \n",
            "                                                                 conv2_block2_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block2_out (Activation)   (None, 56, 56, 256)  0           conv2_block2_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_1_conv (Conv2D)    (None, 56, 56, 64)   16448       conv2_block2_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_1_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block3_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_1_relu (Activation (None, 56, 56, 64)   0           conv2_block3_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_2_conv (Conv2D)    (None, 56, 56, 64)   36928       conv2_block3_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_2_bn (BatchNormali (None, 56, 56, 64)   256         conv2_block3_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_2_relu (Activation (None, 56, 56, 64)   0           conv2_block3_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_3_conv (Conv2D)    (None, 56, 56, 256)  16640       conv2_block3_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_3_bn (BatchNormali (None, 56, 56, 256)  1024        conv2_block3_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_add (Add)          (None, 56, 56, 256)  0           conv2_block2_out[0][0]           \n",
            "                                                                 conv2_block3_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv2_block3_out (Activation)   (None, 56, 56, 256)  0           conv2_block3_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_1_conv (Conv2D)    (None, 28, 28, 128)  32896       conv2_block3_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block1_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_1_relu (Activation (None, 28, 28, 128)  0           conv3_block1_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_2_conv (Conv2D)    (None, 28, 28, 128)  147584      conv3_block1_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block1_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_2_relu (Activation (None, 28, 28, 128)  0           conv3_block1_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_0_conv (Conv2D)    (None, 28, 28, 512)  131584      conv2_block3_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block1_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_0_bn (BatchNormali (None, 28, 28, 512)  2048        conv3_block1_0_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_3_bn (BatchNormali (None, 28, 28, 512)  2048        conv3_block1_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_add (Add)          (None, 28, 28, 512)  0           conv3_block1_0_bn[0][0]          \n",
            "                                                                 conv3_block1_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block1_out (Activation)   (None, 28, 28, 512)  0           conv3_block1_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_1_conv (Conv2D)    (None, 28, 28, 128)  65664       conv3_block1_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block2_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_1_relu (Activation (None, 28, 28, 128)  0           conv3_block2_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_2_conv (Conv2D)    (None, 28, 28, 128)  147584      conv3_block2_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block2_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_2_relu (Activation (None, 28, 28, 128)  0           conv3_block2_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block2_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_3_bn (BatchNormali (None, 28, 28, 512)  2048        conv3_block2_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_add (Add)          (None, 28, 28, 512)  0           conv3_block1_out[0][0]           \n",
            "                                                                 conv3_block2_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block2_out (Activation)   (None, 28, 28, 512)  0           conv3_block2_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_1_conv (Conv2D)    (None, 28, 28, 128)  65664       conv3_block2_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block3_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_1_relu (Activation (None, 28, 28, 128)  0           conv3_block3_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_2_conv (Conv2D)    (None, 28, 28, 128)  147584      conv3_block3_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block3_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_2_relu (Activation (None, 28, 28, 128)  0           conv3_block3_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block3_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_3_bn (BatchNormali (None, 28, 28, 512)  2048        conv3_block3_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_add (Add)          (None, 28, 28, 512)  0           conv3_block2_out[0][0]           \n",
            "                                                                 conv3_block3_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block3_out (Activation)   (None, 28, 28, 512)  0           conv3_block3_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_1_conv (Conv2D)    (None, 28, 28, 128)  65664       conv3_block3_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_1_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block4_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_1_relu (Activation (None, 28, 28, 128)  0           conv3_block4_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_2_conv (Conv2D)    (None, 28, 28, 128)  147584      conv3_block4_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_2_bn (BatchNormali (None, 28, 28, 128)  512         conv3_block4_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_2_relu (Activation (None, 28, 28, 128)  0           conv3_block4_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_3_conv (Conv2D)    (None, 28, 28, 512)  66048       conv3_block4_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_3_bn (BatchNormali (None, 28, 28, 512)  2048        conv3_block4_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_add (Add)          (None, 28, 28, 512)  0           conv3_block3_out[0][0]           \n",
            "                                                                 conv3_block4_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv3_block4_out (Activation)   (None, 28, 28, 512)  0           conv3_block4_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_1_conv (Conv2D)    (None, 14, 14, 256)  131328      conv3_block4_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block1_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_1_relu (Activation (None, 14, 14, 256)  0           conv4_block1_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block1_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block1_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_2_relu (Activation (None, 14, 14, 256)  0           conv4_block1_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_0_conv (Conv2D)    (None, 14, 14, 1024) 525312      conv3_block4_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block1_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_0_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block1_0_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block1_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_add (Add)          (None, 14, 14, 1024) 0           conv4_block1_0_bn[0][0]          \n",
            "                                                                 conv4_block1_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block1_out (Activation)   (None, 14, 14, 1024) 0           conv4_block1_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_1_conv (Conv2D)    (None, 14, 14, 256)  262400      conv4_block1_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block2_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_1_relu (Activation (None, 14, 14, 256)  0           conv4_block2_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block2_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block2_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_2_relu (Activation (None, 14, 14, 256)  0           conv4_block2_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block2_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block2_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_add (Add)          (None, 14, 14, 1024) 0           conv4_block1_out[0][0]           \n",
            "                                                                 conv4_block2_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block2_out (Activation)   (None, 14, 14, 1024) 0           conv4_block2_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_1_conv (Conv2D)    (None, 14, 14, 256)  262400      conv4_block2_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block3_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_1_relu (Activation (None, 14, 14, 256)  0           conv4_block3_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block3_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block3_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_2_relu (Activation (None, 14, 14, 256)  0           conv4_block3_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block3_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block3_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_add (Add)          (None, 14, 14, 1024) 0           conv4_block2_out[0][0]           \n",
            "                                                                 conv4_block3_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block3_out (Activation)   (None, 14, 14, 1024) 0           conv4_block3_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_1_conv (Conv2D)    (None, 14, 14, 256)  262400      conv4_block3_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block4_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_1_relu (Activation (None, 14, 14, 256)  0           conv4_block4_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block4_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block4_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_2_relu (Activation (None, 14, 14, 256)  0           conv4_block4_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block4_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block4_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_add (Add)          (None, 14, 14, 1024) 0           conv4_block3_out[0][0]           \n",
            "                                                                 conv4_block4_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block4_out (Activation)   (None, 14, 14, 1024) 0           conv4_block4_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_1_conv (Conv2D)    (None, 14, 14, 256)  262400      conv4_block4_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block5_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_1_relu (Activation (None, 14, 14, 256)  0           conv4_block5_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block5_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block5_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_2_relu (Activation (None, 14, 14, 256)  0           conv4_block5_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block5_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block5_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_add (Add)          (None, 14, 14, 1024) 0           conv4_block4_out[0][0]           \n",
            "                                                                 conv4_block5_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block5_out (Activation)   (None, 14, 14, 1024) 0           conv4_block5_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_1_conv (Conv2D)    (None, 14, 14, 256)  262400      conv4_block5_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_1_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block6_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_1_relu (Activation (None, 14, 14, 256)  0           conv4_block6_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_2_conv (Conv2D)    (None, 14, 14, 256)  590080      conv4_block6_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_2_bn (BatchNormali (None, 14, 14, 256)  1024        conv4_block6_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_2_relu (Activation (None, 14, 14, 256)  0           conv4_block6_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_3_conv (Conv2D)    (None, 14, 14, 1024) 263168      conv4_block6_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_3_bn (BatchNormali (None, 14, 14, 1024) 4096        conv4_block6_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_add (Add)          (None, 14, 14, 1024) 0           conv4_block5_out[0][0]           \n",
            "                                                                 conv4_block6_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv4_block6_out (Activation)   (None, 14, 14, 1024) 0           conv4_block6_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_1_conv (Conv2D)    (None, 7, 7, 512)    524800      conv4_block6_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block1_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_1_relu (Activation (None, 7, 7, 512)    0           conv5_block1_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_2_conv (Conv2D)    (None, 7, 7, 512)    2359808     conv5_block1_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block1_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_2_relu (Activation (None, 7, 7, 512)    0           conv5_block1_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_0_conv (Conv2D)    (None, 7, 7, 2048)   2099200     conv4_block6_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block1_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_0_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block1_0_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_3_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block1_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_add (Add)          (None, 7, 7, 2048)   0           conv5_block1_0_bn[0][0]          \n",
            "                                                                 conv5_block1_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block1_out (Activation)   (None, 7, 7, 2048)   0           conv5_block1_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_1_conv (Conv2D)    (None, 7, 7, 512)    1049088     conv5_block1_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block2_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_1_relu (Activation (None, 7, 7, 512)    0           conv5_block2_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_2_conv (Conv2D)    (None, 7, 7, 512)    2359808     conv5_block2_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block2_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_2_relu (Activation (None, 7, 7, 512)    0           conv5_block2_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block2_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_3_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block2_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_add (Add)          (None, 7, 7, 2048)   0           conv5_block1_out[0][0]           \n",
            "                                                                 conv5_block2_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block2_out (Activation)   (None, 7, 7, 2048)   0           conv5_block2_add[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_1_conv (Conv2D)    (None, 7, 7, 512)    1049088     conv5_block2_out[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_1_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block3_1_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_1_relu (Activation (None, 7, 7, 512)    0           conv5_block3_1_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_2_conv (Conv2D)    (None, 7, 7, 512)    2359808     conv5_block3_1_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_2_bn (BatchNormali (None, 7, 7, 512)    2048        conv5_block3_2_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_2_relu (Activation (None, 7, 7, 512)    0           conv5_block3_2_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block3_2_relu[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_3_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block3_3_conv[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_add (Add)          (None, 7, 7, 2048)   0           conv5_block2_out[0][0]           \n",
            "                                                                 conv5_block3_3_bn[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "conv5_block3_out (Activation)   (None, 7, 7, 2048)   0           conv5_block3_add[0][0]           \n",
            "==================================================================================================\n",
            "Total params: 23,587,712\n",
            "Trainable params: 23,534,592\n",
            "Non-trainable params: 53,120\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B571TxNMebs_"
      },
      "source": [
        "###Head Architecture###\n",
        "What we have now is a ResNet base model without any output layers. Thus, we must construct this part of the model, which will be placed on top of the base model. The architecture of this is completely up to you! Be cognizant of the activation functions you use and the order of your layers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9B6da8LIehJ-"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import AveragePooling2D\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Flatten\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Conv2D\n",
        "from tensorflow.keras.layers import BatchNormalization\n",
        "from tensorflow.keras.layers import LeakyReLU\n",
        "\n",
        "#TODO: complete architecture of your model\n",
        "#Hint: think about the output dimension.\n",
        "\n",
        "head = base.output\n",
        "head = Flatten()(head)\n",
        "head = Dense(512, activation='relu')(head)\n",
        "head = BatchNormalization()(head)\n",
        "head = Dropout(0.2)(head)\n",
        "head = Dense(256, kernel_regularizer=tf.keras.regularizers.l2(0.01))(head)\n",
        "head = LeakyReLU(alpha=0.05)(head)\n",
        "head = BatchNormalization()(head)\n",
        "head = Dense(2, activation='softmax')(head)   "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnPX4oJ9envZ"
      },
      "source": [
        "Now that we have the base model and the head model, we can combine them into one model. This model will be the model we use to train."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QLnZSsnqelKN"
      },
      "source": [
        "from tensorflow.keras.models import Model\n",
        "\n",
        "model = Model(inputs=base.input, outputs=head)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eW82dDTceriO"
      },
      "source": [
        "###Freeze Base Layers###\n",
        "We want to ensure that our pretrained ResNet weights from imagenet are not changed during training. Thus, we can freeze those weights specifically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hTSg-mYDeudx"
      },
      "source": [
        "#TODO: freeze base model weights\n",
        "for layer in base.layers:\n",
        "  layer.trainable = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nw9zNjpcewtl"
      },
      "source": [
        "Let's now compile the model. Choose a loss function that you think would work best for this task, as well as an appropriate optimizer. The metric you should specify is accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_NJBceqJe0mJ"
      },
      "source": [
        "from tensorflow.keras.optimizers import Adam\n",
        "#TODO: compile model\n",
        "model.compile(optimizer = Adam(lr=LR), \n",
        "              loss = 'binary_crossentropy', \n",
        "              metrics = ['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0SoyFWCZeLe0"
      },
      "source": [
        "### Data augmentation\n",
        "The last thing we need to do before we can start training our model is to initialize our augmentation object for our training set, which will make our model more robust. Specify your desired rotation range and fill mode using ImageDataGenerator from Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kzi3yvZYeSOy"
      },
      "source": [
        "#TODO: define ImageDataGenerator\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "trainAug = ImageDataGenerator(rotation_range = 40,\n",
        "                              width_shift_range = 0.2,\n",
        "                              height_shift_range = 0.2,\n",
        "                              zoom_range = 0.2,\n",
        "                              fill_mode=\"nearest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ataiZOiKe5du"
      },
      "source": [
        "###Training\n",
        "We can now train the model. Because we are utilizing Keras's ImageDataGenerator, use model.fit_generator as opposed to model.fit.\n",
        "\n",
        "If training is taking a while, check to make sure your runtime type is set to \"GPU\", which can be found in the toolbar: \n",
        "\n",
        "Runtime -> Change runtime type -> Hardware Accelerator: GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MpiC-2O3e780",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7bbf59e5-3a6e-4fc9-8f74-f317820d69a5"
      },
      "source": [
        "#TODO: train model\n",
        "\n",
        "callback = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\", \n",
        "                                 patience=8, \n",
        "                                 verbose=0, \n",
        "                                 mode=\"auto\", \n",
        "                                 restore_best_weights=True,)\n",
        "\n",
        "H = model.fit(trainAug.flow(trainX, trainY, batch_size=BATCH_SIZE),\n",
        "              steps_per_epoch=len(trainX)//BATCH_SIZE,\n",
        "              validation_data = (testX, testY),\n",
        "              validation_steps = 20,\n",
        "              epochs = EPOCHS,\n",
        "              callbacks=[callback],\n",
        "              verbose = 2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "109/109 - 57s - loss: 3.5367 - accuracy: 0.8183 - val_loss: 3.6135 - val_accuracy: 0.7581\n",
            "Epoch 2/20\n",
            "109/109 - 19s - loss: 2.9003 - accuracy: 0.8545 - val_loss: 3.4180 - val_accuracy: 0.5903\n",
            "Epoch 3/20\n",
            "109/109 - 19s - loss: 2.4391 - accuracy: 0.8626 - val_loss: 3.5876 - val_accuracy: 0.5839\n",
            "Epoch 4/20\n",
            "109/109 - 19s - loss: 2.0987 - accuracy: 0.8534 - val_loss: 2.0315 - val_accuracy: 0.8516\n",
            "Epoch 5/20\n",
            "109/109 - 19s - loss: 1.8283 - accuracy: 0.8643 - val_loss: 1.7340 - val_accuracy: 0.9290\n",
            "Epoch 6/20\n",
            "109/109 - 19s - loss: 1.6041 - accuracy: 0.8729 - val_loss: 1.5500 - val_accuracy: 0.9194\n",
            "Epoch 7/20\n",
            "109/109 - 19s - loss: 1.4164 - accuracy: 0.8787 - val_loss: 2.9403 - val_accuracy: 0.5806\n",
            "Epoch 8/20\n",
            "109/109 - 19s - loss: 1.2822 - accuracy: 0.8844 - val_loss: 1.9518 - val_accuracy: 0.5323\n",
            "Epoch 9/20\n",
            "109/109 - 20s - loss: 1.1407 - accuracy: 0.8959 - val_loss: 1.5215 - val_accuracy: 0.6935\n",
            "Epoch 10/20\n",
            "109/109 - 19s - loss: 1.0609 - accuracy: 0.8896 - val_loss: 1.1822 - val_accuracy: 0.7871\n",
            "Epoch 11/20\n",
            "109/109 - 19s - loss: 0.9985 - accuracy: 0.8867 - val_loss: 0.9642 - val_accuracy: 0.9323\n",
            "Epoch 12/20\n",
            "109/109 - 19s - loss: 0.9252 - accuracy: 0.8879 - val_loss: 1.7567 - val_accuracy: 0.5871\n",
            "Epoch 13/20\n",
            "109/109 - 19s - loss: 0.8806 - accuracy: 0.8815 - val_loss: 1.0382 - val_accuracy: 0.7677\n",
            "Epoch 14/20\n",
            "109/109 - 19s - loss: 0.8070 - accuracy: 0.8936 - val_loss: 0.9260 - val_accuracy: 0.7903\n",
            "Epoch 15/20\n",
            "109/109 - 19s - loss: 0.7615 - accuracy: 0.8994 - val_loss: 5.9506 - val_accuracy: 0.5774\n",
            "Epoch 16/20\n",
            "109/109 - 19s - loss: 0.7208 - accuracy: 0.8982 - val_loss: 2.8180 - val_accuracy: 0.4387\n",
            "Epoch 17/20\n",
            "109/109 - 19s - loss: 0.6825 - accuracy: 0.8994 - val_loss: 1.6957 - val_accuracy: 0.5548\n",
            "Epoch 18/20\n",
            "109/109 - 19s - loss: 0.6728 - accuracy: 0.8913 - val_loss: 0.5629 - val_accuracy: 0.9452\n",
            "Epoch 19/20\n",
            "109/109 - 19s - loss: 0.6175 - accuracy: 0.9011 - val_loss: 0.9466 - val_accuracy: 0.7581\n",
            "Epoch 20/20\n",
            "109/109 - 19s - loss: 0.6246 - accuracy: 0.8833 - val_loss: 0.5084 - val_accuracy: 0.9516\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-3I-K36we9tb"
      },
      "source": [
        "### Predictions\n",
        "Congrats! You've trained the model. Now let's look at some important metrics to see how well your model performs on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6mCKD5H_fEqi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5bbcb671-dea4-4888-8986-51a6e354994a"
      },
      "source": [
        "from sklearn.metrics import classification_report\n",
        "\n",
        "predIdxs = np.argmax(model.predict(testX, batch_size=BATCH_SIZE), axis=1)\n",
        "\n",
        "print(classification_report(testY.argmax(axis=1), predIdxs,\n",
        "\ttarget_names=lb.classes_))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       covid       0.95      0.93      0.94       136\n",
            "      normal       0.95      0.97      0.96       174\n",
            "\n",
            "    accuracy                           0.95       310\n",
            "   macro avg       0.95      0.95      0.95       310\n",
            "weighted avg       0.95      0.95      0.95       310\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JXxRMQ6yfcLU"
      },
      "source": [
        "We can also compute the confusion matrix to find the sensitivity and specificity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VfnVOJ0Vff2Z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9ecf50ab-534e-4b0c-d673-efca2896e72f"
      },
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "\n",
        "cm = confusion_matrix(testY.argmax(axis=1), predIdxs)\n",
        "print(cm)\n",
        "\n",
        "#TODO: compute sensitivity and specificity from confusion matrix\n",
        "tn, fp, fn, tp = cm.ravel()\n",
        "sensitivity = tp/(fn+tp)\n",
        "specificity = tn/(fp+tn)\n",
        "print(\"sensitivity: {:.4f}\".format(sensitivity))\n",
        "print(\"specificity: {:.4f}\".format(specificity))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[127   9]\n",
            " [  6 168]]\n",
            "sensitivity: 0.9655\n",
            "specificity: 0.9338\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XqTP9N8afxir"
      },
      "source": [
        "### Visualization\n",
        "Let's plot your loss and accuracy metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_5E3isrrf0Rs",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "outputId": "3d352b5d-cf82-4941-dfef-9c891fe8331d"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "N = EPOCHS\n",
        "plt.style.use(\"ggplot\")\n",
        "plt.figure()\n",
        "plt.plot(np.arange(0, N), H.history[\"accuracy\"], label=\"Train Accuracy\")\n",
        "plt.plot(np.arange(0, N), H.history[\"val_accuracy\"], label=\"Validation Accuracy\")\n",
        "plt.plot(np.arange(0, N), H.history[\"loss\"], label=\"Train Loss\")\n",
        "plt.plot(np.arange(0, N), H.history[\"val_loss\"], label=\"Validation Loss\")\n",
        "plt.title(\"Loss and Accuracy Plot\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')\n",
        "plt.savefig(\"midterm_plot.png\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAAEaCAYAAABqw5/dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde1zUdb748ddcgOEOgqCgAt4VFMW85A3Ee17SbLfsslpruVnWnt2Hp7ba2rbTrrW1bZ5jvzrVZqet7GJq5TXURSVNBfECgYCICl4QL1yGAWbm+/uDnVlQlNsMM8y8n49Hj0cz3/l+v+8ZUd7zubzfKkVRFIQQQgghALWjAxBCCCGE85DEQAghhBBWkhgIIYQQwkoSAyGEEEJYSWIghBBCCCtJDIQQQghhJYmBcDpr1qxBq9U6OgyXpVKp+Mc//uHoMIQQTkoSg05i8eLFTJkyxdFhOJ0ff/wRjUbDyJEjHR2KU1i8eDEqlQqVSoVWqyUqKopf/epXlJWVtfmae/fuRaVScerUKdsFKoRwWpIYiE7t3Xff5bHHHqOgoIDMzExHh4OiKNTV1Tk0hgkTJnDu3DlOnTrFqlWrWLduHb/4xS8cGpMQovOQxMBF5ObmMmvWLPz8/PDz82POnDnk5+dbj5eXl/PQQw/RrVs3vLy86NmzJ7/5zW+sx/fu3cu4cePw9/fH39+f+Ph4tm3bdtP7FRYWctdddxEREYGPjw9Dhgzh448/bvSapKQklixZwssvv0y3bt3o0qULv/jFL6isrLS+xmw28/vf/56wsDD8/Py45557uHLlSove87Vr1/j8889ZunQp99xzD+++++4NrykoKODuu++mS5cu+Pj4MHToUL777jvr8fT0dGbMmEFAQAB+fn6MGjWKH3/8EYA//OEP9O3bt9H1rv/2bJn22LVrF8OHD8fLy4uUlJQWfT4Aq1evZvDgwXh5eREWFsaCBQus9x4wYMANr3/44YeZPHnyLT8XT09PunXrRo8ePbjzzjv59a9/zdatW6murm7y9efOnePee+8lKCgIb29vkpKSOHToEACnTp1iwoQJAMTExKBSqUhKSrrl/YUQnZskBi6gurqaadOmYTAYSE1NJTU1lcrKSmbMmEFtbS0Azz//PBkZGWzcuJG8vDw+//xzBg0aBIDRaGTu3LmMHj2ajIwMMjIy+MMf/oCPj89N71lZWUlycjJbtmzh2LFjPProozz00EPs2rWr0eu++uorLl++zD//+U/Wrl3Ld999x6uvvmo9/t///d/89a9/5S9/+QsZGRmMGDGCl156qUXv+x//+AcDBw5kyJAhLF68mE8++YSqqirr8fPnzzN27FiuXr3KN998w7Fjx3j55ZdRq+t/7LOyspg4cSLBwcHs3LmTw4cP8x//8R+YzeaWffD/Yjabefrpp/nrX/9KTk4Ot912W4s+nxdffJGnn36aZcuWcezYMbZu3UpCQgIAS5YsoaCggNTUVOvrKyoq+OKLL3j00UdbFZ+3tzdmsxmj0XjDMUVRmDdvHjk5OXz33XccOHCA8PBwpk6dyqVLl+jZsycbN24E4MCBA5w7d46vv/66VfcXQnQyiugUFi1apEyePLnJY++//77i7e2tlJaWWp87f/68otPplI8++khRFEWZO3eusmjRoibPv3z5sgIou3btaleMc+fOVZYsWWJ9nJiYqAwdOrTRa371q18pY8aMsT6OjIxUnn322UavWbBggaLRaJq9X3x8vLJq1Srr4wEDBijvvfee9fHzzz+vhIeHK5WVlU2e/8ADDyhDhw5VTCZTk8dffPFFpU+fPo2e27NnjwIohYWFiqIoyocffqgAyu7du5uNt+HnU1lZqeh0OuUvf/nLTV8/Z84c5f7777c+fuedd5TQ0FClpqbmpudc/3OSlZWl9O7dWxk9erT1OUD5+OOPFUVRlJSUFAVQsrKyrMcNBoPSrVs35aWXXmryPQshXJuMGLiArKwsBg8eTGhoqPW58PBwBgwYQFZWFgDLli3jq6++Ii4ujqeeeootW7ZYvxkHBwezZMkSpk+fzsyZM1m5ciW5ubm3vKder+eZZ54hNjaWLl264Ofnx+bNmykqKmr0uvj4+EaPIyIiuHDhAlA/vVFcXMzYsWMbvWb8+PHNvucff/yRn376ifvuu8/63KJFixpNJ6SnpzN27Fh8fX2bvEZ6ejqTJ0+2jiC0x/WLH5v7fLKysjAYDEybNu2m11y6dCnr1q2zTq289957LFq0CE9Pz1vG8s9//hM/Pz+8vb2Ji4ujd+/efPrpp02+Nisri5CQEAYPHmx9zsvLi9GjR1t/doQQ7kUSAzcxffp0Tp8+zXPPPYfBYOCBBx4gOTkZk8kE1P/SSU9PZ+rUqaSmphIXF9fknL3FihUr+Mc//sGLL77Irl27yMzM5I477rBOXVhc/0tMpVK1eqi+Ke+++y61tbWEh4ej1WrRarU8//zzHDp0yGaLENVqNcp1zUebWlio0WjQ6XSNnmvp53MrM2fOJCwsjI8//pjMzEzS09N55JFHmj1v9OjRZGZm8tNPP2EwGPj+++/p3bt3i+8rhHBvkhi4gNjYWLKzs7l06ZL1uQsXLpCbm0tcXJz1uS5durBw4ULeffddNm3aRGpqKtnZ2dbjcXFx/OY3v2HLli388pe/5H//939ves/du3dz//338/Of/5z4+Hh69+7NiRMnWhV3QEAAkZGR/PDDD42eT0tLu+V5lkWHq1evJjMz0/rfkSNHmDhxojWhGTFiBD/88EOjdQcNjRgxgh07dtw0UQkLC+PixYvW5AkgIyOjRe+tuc9n8ODB6HQ6tm/fftNrqNVqHnnkEd577z3ee+89Jk6c2OSCxOt5e3vTt29foqOjmx1diI2NpaysrNHPQU1NDT/++KP1Z8dyjYafgxDCdUli0IlUVlY2+kWYmZlJTk4O9913H127duWee+4hIyOD9PR07r33XiIjI7nnnnsAeO655/j666/Jzc0lLy+PTz75BD8/P3r16kV+fj5PP/00e/fupaioiH379rFnz55Gw8vXGzBgABs3buTAgQNkZ2fz6KOPUlJS0ur39Nvf/pa33nqLjz/+mLy8PN544w1SUlJuec4//vEP1Go1Dz30EHFxcY3+u//++62LEJctW4bZbObOO+8kLS2NwsJCvvvuO7Zs2QLAf/7nf5KXl8f999/PoUOHKCgo4Msvv2Tfvn0ATJo0Cb1ezwsvvGA9tnr16ha9r+Y+Hz8/P37729/yhz/8gdWrV3PixAmOHDnCn//850bX+eUvf0lOTg7vv/9+qxcdtkRycjKjRo3ivvvuIy0tjePHj/OLX/wCg8HAY489BkBUVBRqtZrNmzdz8eJFrl27ZvM4hBBOxNGLHETLLFq0SAFu+G/AgAGKoihKTk6OMnPmTMXX11fx9fVVZs2apeTl5VnP/+Mf/6jExsYqvr6+SkBAgDJx4kRlz549iqIoSklJiTJ//nwlMjJS8fT0VLp3764sWbJEuXr16k3jOX36tDJt2jTFx8dH6datm/LCCy8oDz/8sJKYmGh9TWJiovLLX/6y0Xkvv/yyEhUVZX1sMpmU3/3ud0pISIji4+OjLFiwQPnrX/96y8WH8fHxyr333tvksdLSUkWr1VoXIebm5irz5s1TAgICFG9vb2Xo0KHKpk2brK//8ccflcmTJys+Pj6Kn5+fMnr0aOXHH3+0Hv/ggw+UmJgYRafTKTNmzFA+++yzGxYfNhVrSz4fs9ms/O1vf1P69++veHh4KGFhYcrdd999w7XmzZundOnSRTEYDDf9TCxutUjVggaLDxWl/s//nnvuUQIDAxWdTqdMnDhROXjwYKNzXn31VSUiIkJRq9WN3oMQwvWoFOW6SVQhhFMZNWoU48aN480333R0KEIINyAF6YVwUpcuXeK7774jIyODtWvXOjocIYSbkMRACCfVtWtXgoODWbVqlewqEEJ0GEkMhHBSMssnhHAE2ZUghBBCCCtJDIQQQghhZbephLbsaQcIDQ1tVKjH2Uh87SPxtY/E1z7OHF9ERISjQxACkBEDIYQQQjQgiYEQQgghrCQxEEIIIYRVi9YYVFVV8c4773DmzBlUKhWPPfYY/fv3t3dsQggh2klRFEpLS5vsDCrcl4eHB127dkWlUt1wrEWJwYcffsiwYcP47W9/i9FopKamxuZBCiGEsL3S0lKMRmOznTaFe6mrq6O0tJSwsLAbjjU7laDX6/npp59ITk4GQKvV4uvra/sohRBC2FxdXR0eHh6ODkM4GQ8Pj5uOIjXbROnUqVO8++679OjRg6KiInr37s3ixYvR6XSNXpeSkmJtl7ty5Upqa2vbFKxWq8VoNLbp3I4g8bWPxNc+El/7OHN89vpGX1xcLKMFokm1tbVERkbe8HyzUwkmk4nCwkIefvhh+vXrx4cffsiGDRu49957G71uypQpTJkyxfq4rXuFnXmfMUh87SXxtY8947PMQ7dnP707f37tJXUMhLNodiohJCSEkJAQ+vXrB8CYMWMoLCy0e2BCiI71ww8/WEf9hLCVy5cvM2nSJCZNmkRsbCxDhw61Pm5uZDkzM5Nnn3221fc8duwYYWFh7Ny5s61hu7VmRwyCgoIICQmhpKSEiIgIjh07Ro8ePToiNiFEB6qqquLatWuYTCY0Go2jwxEuokuXLuzatQuA1157DV9fXx5//HHrcaPRiFbb9K+iYcOGMWzYsFbfc/369YwePZqvv/7auj7OHlz170qLdiU8/PDDrFq1CqPRSFhYGMuWLbN3XEKIDlZdXY2iKFRUVBAUFOTocIQLW758OV5eXhw/fpyRI0cyf/58nnvuOWpqatDpdKxatYq+ffuSlpbG22+/zSeffMJrr71GcXExRUVFnD17lqVLl/LII4/ccG1FUfjmm2/48ssvmTt3LgaDwbombtWqVaxbtw6VSsXkyZP5/e9/z8mTJ1mxYgVlZWVoNBref/99SkpKrPcFeOaZZxg2bBj33nsvI0aM4M477yQ1NZUnnniCyspKPv74Y2pra4mJiWH16tX4+Phw8eJFVqxYQVFREVCfFO3cuZPg4GCWLl0KwJ/+9CdCQ0N59NFHO+iTb5kWJQbR0dGsXLnS3rEIIRxEURT0ej0AV69elcTARdV98g7m0ydtek11r9543P+rVp937tw5Nm3ahEajoaKigm+//RatVktqaiqvvPIKH3744Q3n5OXlsX79eiorKxk7diyLFy++YcfFgQMHiIqKIiYmhnHjxvH9998zZ84cduzYwdatW9myZQs+Pj5cuXIFgGXLlrF8+XJmzZqFwWDAbDY32+snODiYHTt2APVTJQ8++CAAf/7zn/n0009ZsmQJzz33HGPHjuWjjz7CZDJRVVVFt27deOihh1i6dClms5n169ezbdu2Vn929ma3JkpCiM7DYDBg2aB07do1B0cj3MGcOXOsw/Dl5eU88cQTFBYWolKpbrqNburUqXh5eeHl5UVoaCilpaU3LNpcv3498+bNA2DevHl88cUXzJkzh9TUVBYuXIiPjw9Q/8u9srKSc+fOMWvWLIAbdtvdjOX6ADk5Ofz5z3+mvLycqqoqkpKSANi7dy//8z//A4BGoyEgIICAgACCg4M5duwYpaWlDBkyhC5durTwE+s4khgIIayjBVA/YiBcU1u+2dtLw3o4K1euZPz48Xz00UecPn2a+fPnN3lOw22XGo3mhq2nJpOJ7777jq1bt/K3v/0NRVG4fPkylZWVrYpNo9FgNputj68v6mdJLgCefPJJ1qxZQ1xcHGvXriUtLe2W137ggQdYu3YtFy9e5L777mtVXB1FeiUIISQxEA5VXl5Ot27dAFi7dm2br7N7924GDx5MZmYm6enpZGRkMHv2bDZt2kRSUhKfffaZ9Wf9ypUr+Pn5ERERwebNm4H6BECv19OzZ09OnDhBTU0N165dY8+ePTe9Z2VlJeHh4dTV1fHVV19Zn58wYQJr1qwB6hOW8vJyAO644w527tzJ4cOHmTRpUpvfqz1JYiCEoLq6GqhfQS5TCaKjPfHEE7zyyiskJydjMpnafJ3169dzxx13NHpu9uzZrF+/nuTkZGbMmMG0adOYNGkSb7/9NgCrV6/m/fffJzExkVmzZnHx4kUiIyOZO3cuEydOZMmSJcTFxd30nk8//TQzZ85k9uzZ1m39AP/1X/9FWloaiYmJTJkyhdzcXKB+1GPcuHHceeedTrujodnKh23V3OKNm3HmAiQg8bWXxNc+9oovMzOT3bt3Exsby08//cSyZctQq1v/vcFdPz9bsFeBI6l86FzMZjOTJ0/mgw8+oHfv3g6N5WaVD2XEQAiBXq9HpVLRrVs3zGazddhTCGE7ubm5jBo1igkTJjg8KbgVWXwohECv1+Pt7W3dpihbFoWwvQEDBnDo0CFHh9EsGTEQQqDX6/Hx8bEmA7LOQAj3JYmBEILq6mp8fHzw8fHBw8NDdiYI4cYkMRBCWKcSVCoVgYGBMmIghBuTxEAIN2cph2wp2hIUFCQjBkK4MUkMhHBztbW1mEymRolBeXl5o8pvQrTV/Pnzb2h//O6777JixYqbnjNv3jwyMzMBWLhwYZMjWK+99hqrV6++5b03b95srR8A9RUWU1NTWxP+LT3//PMMHTrU5f6uSGIghJuzFDeyJAaBgYGyZVHYzPz589mwYUOj5zZs2MBdd93VovM/++wzAgMD23TvLVu2cOLECevjZ555hsTExDZd63pms5nNmzcTGRnJDz/8YJNrNuX6ss8dQRIDIdycpUSst7c3gOxMEDY1Z84cUlJSqK2tBeD06dOcP3+eMWPGsGLFCqZOncqECRN49dVXmzx/xIgRlJWVAfDmm28yZswYZs+eTUFBgfU1H3/8MdOmTSMpKYmHHnoIvV7PgQMH2LZtGy+99BKTJk2isLCQ5cuX8+233wL15ZOTk5NJTEzkqaeesvZDGDFiBK+++iqTJ08mMTGRvLy8JuNKS0tjwIABLFq0iPXr11ufv3jxIosWLSIpKYmkpCQOHDgAwOeff05iYiJJSUksW7YMoFE8UN/J2HLtOXPm8OCDDzJ+/HgAfvGLXzBlyhQmTJjA//3f/1nP2blzJ5MnTyYpKYkFCxZgNpsZPXq0tZCX2Wxm1KhRrSrsJXUMhHBzlsSg4VQC1NcyiIqKclhcwvbe/bGEk5cNNr1m7y46lo6+edXG4OBghg8fzo4dO5g5cyYbNmxg7ty5qFQqnn32WYKDgzGZTCxYsICsrCxiY2ObvM6RI0fYsGEDO3fuxGQyMXnyZIYOHQrArFmzmmx9PH36dKZNm8acOXMaXctgMPDkk0+ybt06+vTpw+OPP86aNWtYunQpACEhIezYsYO///3vvP3227z55ps3xLN+/Xrmz5/PzJkz+dOf/kRdXR0eHh5NtlvOycnhzTffZNOmTYSEhFhbPt/KsWPHSE1Ntf4dfOuttwgODqa6uprp06cze/ZszGYzv/nNb9i4cSNRUVFcuXIFtVrN3Xffzbp161i6dCmpqanExsYSGhra7D0tZMRACDd3fWJg2bIoIwbCVhpOJ6xfv946jbBx40YmT55McnIyubm5jYb9r7d//35mzpyJj48P/v7+TJ8+3XosJyeHOXPmkJiYyLp168jJybllPPn5+fTq1Ys+ffoAcM8997Bv3z7rcUsb5vj4eE6fPn3D+bW1taSkpHDHHXfg7+9PQkICu3btAurbLS9evBj4d7vlvXv3MnfuXEJCQoD6ZKk5w4cPb5SYv/feeyQlJTFz5kyKi4s5efIk6enpjBkzxvo6y3Xvu+8+vvjiC6B+KmbhwoXN3q8hGTEQws1Z1hhYphIsWxZlZ4LrudU3e3uaMWMGL7zwAkePHqW6upr4+HiKiop4++232b59O0FBQSxfvvyG9sYt1drWx82x9JbQaDRNNnXatWsX5eXl1vUK1dXV6HQ6pk2b1qr7aLVa68JFs9lMXV2d9VjD1s5paWns3r2bzZs34+Pjw7x58275WUVGRtK1a1f27NlDRkYG/+///b9WxSUjBkK4Ob1ej06na9TpTbYsClvy8/Nj3LhxPPXUU8yfPx+AiooKfHx8CAgI4OLFi+zYseOW17j99tvZsmUL1dXVVFZWsn37duuxm7U+9vPzo7Ky8oZr9e3blzNnznDy5EkAvvzyS8aOHdvi97N+/Xr++te/kp6eTnp6OgcPHmT37t3o9fom2y2PHz+eb775hsuXLwNYpxJ69uzJkSNHANi6dWujxKCh8vJygoKC8PHxIS8vj/T0dKB+PcT+/fspKipqdF2A+++/n2XLljF37txWd3GUxEAIN2cpbtRQYGCgbFkUNjV//nyysrKs0whxcXEMGTKEsWPH8thjjzFq1Khbnj906FDmzZvHpEmTuPfeexk+fLj12M1aH8+bN4/Vq1eTnJxMYWGh9XmdTsdbb73FkiVLSExMRK1Ws2jRoha9D71ez86dO5k6dar1OV9fX0aNGsX27dubbLc8cOBAfv3rXzNv3jySkpJ44YUXAHjggQfYt28fSUlJHDp0qNEoQUPJyckYjUbGjRvHyy+/zIgRI4D6bqFvvPEGDz30EElJSTzyyCPWc2bMmEFVVVWrpxFA2i63msTXPhJf+9gjvi+//BK1Ws2CBQusz2VlZbFjxw4WLVrUqq1i7vj52Yq0XRa2lJmZye9///tGux6uJ22XhRBNsvRJaKjhzgQhROeyatUqHn74YZ577rk2nS+JgRBurmE5ZAtJDITovJ588kkyMjIYM2ZMm86XxEAIN2Y0Gqmtrb0hMZAti0K4L0kMhHBj11c9tJAti0K4L0kMhHBj1/dJaEgSAyHckyQGQrix66seNiRdFoVwT5IYCOHGmksMzGYzFRUVHR2WcCGXL19m0qRJTJo0idjYWIYOHWp9bGmsdDOZmZk8++yzrbpfw6ZLom1aVBL58ccfR6fToVar0Wg0rFy50t5xCSE6QHOJAdTvTGhr21shunTpYu0j8Nprr+Hr68vjjz9uPW40GtFqm/5VNGzYMIYNG9YhcYp/a3GvhBdffJGAgAB7xiKE6GB6vR4PD48m/2G2JAPSZVHY2vLly/Hy8uL48eOMHDmS+fPn89xzz1FTU4NOp2PVqlX07duXtLQ03n77bT755BNee+01iouLKSoq4uzZsyxdurRRpb9bOX36NL/+9a8pKysjNDSUt956ix49evDNN9/w+uuvo1arCQgI4JtvviEnJ4ennnqK2tpazGYzH374Ib1797bzJ+JcpImSEG6sqeJGFr6+vmi1Wtmy6EKOHKrk2mWjTa8Z2EVL/G1+rT7v3LlzbNq0CY1GQ0VFBd9++y1arZbU1FReeeUVPvzwwxvOycvLY/369VRWVjJ27FgWL16Mh4dHs/d69tln+fnPf869997Lp59+yrPPPsv//d//8cYbb/D555/TvXt368/5Rx99xCOPPMLdd99NbW1tk02UXF2LE4NXXnkFgKlTpzJlypQbjqekpJCSkgLAypUrW9X7uVFAWm2bz+0IEl/7SHztY+v46urqCAgIuOk1Q0JC0Ov1Lb6nu31+ou3mzJljbe5TXl7OE088QWFhISqV6qbNhKZOnYqXlxdeXl6EhoZSWlraolLShw4dsiYaP/vZz/jjH/8IwMiRI1m+fDl33nmntdXybbfdxt/+9jdKSkqYPXu2240WQAsTg5dffpkuXbpw7do1/uu//ouIiAgGDx7c6DVTpkxplDC0tR65M9cyB4mvvSS+9rF1fNeuXSM4OPim1/Tz86O0tLTF93S3z8+W7NUroaG2fLO3F19fX+v/r1y5kvHjx/PRRx9x+vRpawfG6zXs+aDRaDAa2zf68frrr5Oens7333/P1KlT+f7771mwYAEJCQmkpKSwcOFCXn/9dSZMmNCu+3Q2LdqV0KVLF6B+znHkyJHk5+fbNSghRMdoqhxyQ0FBQVy7dk22LAq7Ki8vp1u3bgCsXbvW5tcfOXIk69evB2DdunWMHj0agMLCQkaMGMEzzzxDSEgIxcXFnDp1iujoaB555BFmzJhBdna2zeNxds2OGBgMBhRFwdvbG4PBwNGjR7n77rs7IjYhhB2ZzWYMBsMNVQ8barhlUXYmCHt54oknWL58OW+++WajdsZtlZSUhFpd/7137ty5/OlPf+Kpp55i9erV1sWHAC+99BKFhYUoisKECROIi4vjv//7v/nyyy/RarWEhYXx61//ut3xdDbNtl2+cOECr7/+OgAmk4nx48db+2nfirRddgyJr33cKb6qqio++OADkpKSGDp0aJOvOXv2LF9//TV33nlni3YmuNPnZ2vSdll0tJu1XW52xCA8PJy//OUvdglKCOE4t6phYGGpZSA7E4RwH1L5UAg31ZLEwLJlUXomCOE+JDEQwk21JDFQqVQEBQVJYiCEG5HEQAg3ZemseKvFh1C/G0mmEoRwH5IYCOGm9Ho9Go2m2YVpsmVRCPciiYEQbspSw0ClUt3ydYGBgdJlUQg3IomBEG6queJGFrIzQbTH/Pnz2blzZ6Pn3n33XVasWHHTc+bNm0dmZiYACxcubPJn77XXXmP16tW3vPfmzZvJzc21Pl65ciWpqamtCb9JaWlp3H///e2+jrOSxEAIN6XX65tdXwCN2y8L0Vrz589nw4YNjZ7bsGFDi+rhAHz22WdtLq61ZcsWTpw4YX38zDPPkJiY2KZruRNJDIRwU7fqrNiQbFkU7TFnzhxSUlKora0F6lsgnz9/njFjxrBixQqmTp3KhAkTePXVV5s8f8SIEZSVlQHw5ptvMmbMGGbPnk1BQYH1NR9//DHTpk0jKSmJhx56CL1ez4EDB9i2bRsvvfQSkyZNorCwkOXLl/Ptt98CsHv3bpKTk0lMTOSpp56ipqbGer9XX32VyZMnk5iYSF5eXovf69dff01iYiITJ060NmoymUwsX76ciRMnkpiYyDvvvAPAe++9x/jx40lMTOTRRx9t5adqX9J2WQg3pChKi6cSVCoVgYGBkhi4gF27dnHx4kWbXjMsLIxJkybd9HhwcDDDhw9nx44dzJw5kw0bNjB37lxUKhXPPvsswcHBmEwmFixYQFZWFrGxsU1e582e/7UAACAASURBVMiRI2zYsIGdO3diMpmYPHmytWLnrFmzePDBBwH485//zKeffsqSJUuYPn0606ZNY86cOY2uZTAYePLJJ1m3bh19+vTh8ccfZ82aNSxduhSo7yq6Y8cO/v73v/P222/z5ptvNvs5nD9/npdffpnvv/+eoKAgfv7zn7N582YiIyM5d+4cu3fvBv49Jbdq1SoOHTqEl5eX003TyYiBEG7I0gOlJYkB/HtnghBt0XA6Yf369dZphI0bNzJ58mSSk5PJzc1tNOx/vf379zNz5kx8fHzw9/dn+vTp1mM5OTnMmTOHxMRE1q1bR05Ozi3jyc/Pp1evXvTp0weAe+65h3379lmPW1owx8fHc/r06Ra9x8OHDzN27FhCQ0PRarUsWLCAffv2ERUVRVFREb/73e/YuXMn/v7+AAwePJjHHnuML7/80tp+2lnIiIEQbshS3KglawygPjEoLCzEbDZbm9OIzudW3+ztacaMGbzwwgscPXqU6upq4uPjKSoq4u2332b79u0EBQWxfPly63B+az355JOsWbOGuLg41q5dS1paWrvitWzh1Wg0mEymdl0rKCiIXbt2sWvXLtasWcPGjRt56623+PTTT9m3bx/btm3jb3/7G6mpqWi1zvErWf6GC+GGWlL1sCHLlsXKykp7hiVclJ+fH+PGjeOpp55i/vz5AFRUVODj40NAQAAXL15kx44dt7zG7bffzpYtW6iurqayspLt27dbj1VWVhIeHk5dXR1fffVVo/s29TPbt29fzpw5w8mTJwH48ssvGTt2bLveY0JCAvv27aOsrAyTycTXX3/N2LFjKSsrQ1EU5syZw+9+9zuOHj2K2WymuLiY8ePH88ILL1BeXk5VVVW77m9LzpGeCCE6lKXqYWumEqB+Z0JAQIDd4hKua/78+SxevJj//d//BSAuLo4hQ4YwduxYIiMjGTVq1C3PHzp0KPPmzWPSpEmEhoYyfPhw67Gnn36amTNnEhISQkJCgjUZmDdvHr/97W957733+OCDD6yv1+l0vPXWWyxZsgSTycSwYcNYtGhRq97Pnj17iI+Ptz5+//33ef7557nrrrtQFIUpU6Ywc+ZMjh8/zlNPPWUtEPb8889jMplYtmwZFRUVKIrCI4884lRtzZttu9xW0nbZMSS+9nGX+DIzM9m9ezePPPJIi6YTKisr+fvf/37LFs22jM9enDk+abssOtrN2i7LVIIQbkiv16NSqdDpdC16vWxZFMJ9SGIghBuyFDdqrhyyhWXLouxMEML1SWIghBtqaQ2DhqT9shDuQRIDIdxQS6seNiRdFjsnDw8P6urqHB2GcDJ1dXV4eHg0eUx2JQjhhvR6vXWnQUs13LIoOxM6j65du1JaWmotSSwE1CeMXbt2bfKYJAZCuJnWlENuSLYsdk4qlYqwsDBHhyE6EZlKEMLN1NbWYjKZWp0YWPZZyzoDIVybJAZCuJnWFjey8PPzQ6vVys4EIVycJAZCuJnW9kmwkC6LQrgHSQyEcDOt7ZPQkGxZFML1SWIghJtpT2JgKXIkWxaFcF2SGAjhZto6lQD1IwbSZVEI1yaJgRBuprq6Gp1Oh0ajafW5sjNBCNfX4sTAbDbzn//5n6xcudKe8Qgh7MzSJ6EtLLUMZGeCEK6rxYnB5s2bm2zPKIToXNpS3MjCz88PjUYjIwZCuLAWJQZlZWVkZGQwefJke8cjhLCz9iQGKpVKdiYI4eJaVBJ5zZo1PPDAA9bCKE1JSUkhJSUFgJUrVxIaGtq2gLTaNp/bESS+9pH42scW8RkMBkJCQtp8nbCwMC5dutTk+e7w+Qnh6ppNDNLT0wkMDKR3795kZWXd9HVTpkxhypQp1seXLl1qU0ChoaFtPrcjSHztI/G1T3vjMxqN1NTUoFKp2nwdb29vysrKuHjxImp140FHV//87CkiIsLRIQgBtCAxyM3N5dChQxw+fJja2lqqq6tZtWoVTz75ZEfEJ4SwofZsVbRouGVRmikJ4XqaTQzuu+8+7rvvPgCysrL49ttvJSkQopNqT3EjC8uWxWvXrkliIIQLkjoGQriRtjZQaqhh+2UhhOtp0eJDi9jYWGJjY+0VixDCzmwxYiBbFoVwbTJiIIQbsUViIF0WhXBtkhgI4Ub0ej0eHh5ota0aLLxBUFCQVD8UwkVJYiCEG6murm7XaIGFJTFQFMUGUQkhnIkkBkK4kfZUPWwoKCgIk8kkXRaFcEGSGAjhRmyVGEiXRSFclyQGQrgRW44YgCQGQrgiSQyEcBNmsxmDwdCuqocWsmVRCNflVInB5s2bSUtLo6amxtGhCOFybFHcyMKyZVF2Jgjhetq3Z8mGampqqK6uZtu2bXh4eDBo0CCGDRtmHbK0J5PJxKlTp8jLy2PAgAHExMTY/Z5CdDRb1DBoSNovC+GanCYx8PLy4s65d2Ey17F7dyrHjx/n6NGjREdHM3z4cHr06IFKpbLpPcvKysjOziYnJ8f6baqyslISA+GS7JEYFBUVoSiKzf9uCiEcx2kSg9oaM3tTKuk/OIipU6cyduxYjh07xrFjx1i/fj0hISEMGzaMAQMGtKs4i8Fg4MSJE2RnZ1vbxsbExDB48GBKSkrIyMigpqYGLy8vG747IRzP1olBYGCgdcuiv7+/Ta4phHA8p0kMPL3UBIVoyDx4Gd8AP4JDfBkzZgy33XYbJ06cIDMzkx07dpCWlsaQIUMYMmQIfn5+Lbq22WzmzJkzZGdnc/LkSUwmE6GhoUycOJH+/ftb/6H08vIiPT2d06dP069fP3u+XSE6nGVUzBaLD6HxzgRJDIRwHU6TGAAMSfDmapnC4f16Jk73R6tVodVqGTx4MIMGDaK4uJjMzEwOHjxIeno6/fr1Y/jw4YSFhTV5vatXr1qnCiorK9HpdMTGxjJ48GC6du16w/Bnt27d8PLyoqioSBID4XL0ej0ajQZPT0+bXK9hYtCzZ0+bXFMI4XhOlRh4eKqZMCWMrRtKyM6sZuht/x7yVKlU9OjRgx49enD16lWOHj1KVlYWubm5REREEB8fT58+fTAajeTn55OdnU1JSQkqlYpevXoxYcIEYmJibjkNoVar6dWrF6dOnZJ5U+FyLDUMbPVzbdmyKDsThHAtTpUYAHSP9KHPQC8KcmoIj/AgPMLjhtcEBQUxceJERo8eTXZ2NkeOHGHLli34+flRU1NDXV0dQUFBjB07loEDB7Z4ygEgOjqavLw8SktLbzoSIURnZKviRhbSZVEI1+R0iQHAgDgdpefryDygJ2mGP166pssteHl5MXz4cOLj4yksLCQrKwsfHx8GDx5M9+7d2/TNKCoqCoCioiJJDIRL0ev1rUqSW0K2LArhepyqwJGFRqMiYYwvxjqFIwf1zXZwU6vV9OnTh7lz5zJlyhQiIiLaPFzq4+NDWFgYp06datP5QjgrW3VWbMhS5Ei6LArhOpwyMQDwD9QwKN6bCyVGTp+s7dB7R0dHc/78eQwGQ4feVwh7URTF5lMJIF0WhXBFTpsYAMT08yQ0XEvW4WoqK0wddt+oqCgUReH06dMddk8h7MlgMKAoil0SA5BmSkK4EqdODFQqFcNH+6DWqDi8X4/Z3DHDleHh4eh0OplOEC7DUtzIVjUMLCyJgexMEMJ1OHViAKDzVjP0Nm+uXjaRl90xQ/uWbYuWcq9CdHa2rnpoIV0WhXA9Tp8YAET09KRHtAcnsmu4fMnYIfeMjo6murqa0tLSDrmfEPZky86KDcmWRSFcT6dIDADiEnzw9lFzeL8eY539v8X36tULQKYThEuw14gBIImBEC6m0yQGHh716w30ejNZh6vtfj8fHx/Cw8MlMRAuQa/Xo1Kp0Ol0Nr92UFCQbFkUwoV0msQAIKSrlr4DvThdWMu5s/bfwhgVFcX58+etw7BCdFZ6vR5vb2+7lPmWLYtCuJZOlRgADIjVERis4cjBagzVZrveKzo6GkC2LYpOzx41DCxky6IQrqXTJQZqjYrhY3wwmRQyDzRfFbE9ZNuicBX2qHpoERgYCMiWRSFcRbO9Empra3nxxRcxGo2YTCbGjBnDz3/+846I7ab8AzTExntzLKOaU/m1xPTzsst9VCoVUVFR1m2L0m1RdFZ6vZ7g4GC7XNvf31+2LArhQppNDDw8PHjxxRfR6XQYjUZeeOEFhg0bRv/+/TsivpuK6uvJhXN1ZB+pJjRci3+Axi73iY6OJjc3lwsXLtCtWze73EMIe7KUQ7Z1cSMLlUpFQECAJAZCuIhmpxIarmQ2mUyYTCan+OasUqmIH+mDVvuvqogm+0wpWLYtFhUV2eX6QthbbW0tJpPJblMJ8O+dCUKIzq9FbZfNZjNPP/0058+fZ/r06fTr1++G16SkpJCSkgLAypUrCQ0NbVtAWm2rzh2f7MPOLec5c1LNiNtD2nTP5vTo0YOzZ88ya9asVsfX0SS+9nHF+MrKyoD6NTP2em/du3fnzJkzqNVql/v8hHA3LUoM1Go1f/nLX6iqquL111/n9OnT1m/SFlOmTGHKlCnWx5cuXWpTQKGhoa061zcAevX25GjGFfyC6gjp2qK31Co9evRg//791vfd1vfWEVr7+TV08uRJfHx87Dpl0p74OoIrxldSUgLUj/jZ6715eXlhNBq5cuUKdXV1drmHLTjzn29ERISjQxACaOWuBF9fX2JjY8nMzLRXPG0SO8wbHz81h/dXUVdr+ymFqKgowLW3LRqNRrZt28a2bdswm+27DVR0LHs1UGrIsjPBMjohhOi8mk0MysvLqaqqAurnKo8ePUpkZKTdA2sNrYeKhNE+GKoVMvZX2Xy9QVhYGN7e3i69zuDMmTPU1dVx7do1CgsLHR2OsCF7lkO2sNQyuHz5st3uIYToGM2Ou1+5coXVq1djNptRFIXbb7+dESNGdERsrRIcqiUuwZtj6dWk79cz4nYf1GrbLJK0bFs8deqUy36bzs/Px9PTE51OR0ZGBn369HF0SMJGOmLEwM/PD7VaTVlZmbUwmBCic2o2MYiKiuK1117riFjaLbqvF2YzZB2uJvNHPcNH+6CyUXIQHR1NTk4OxcXFdv0H1hFMJhOFhYXExMTQrVs3UlNTOXfuHN27d3d0aMIGqqur0el0aDT22dIL9euQAgMDZcRACBfQ6SofNqd3fy8GDdVRfLqOIwerbVYZsVevXqhUKk6cOGGT6zmTkpISDAYDffv2ZdCgQXh5eXH48GFHhyVsxJ41DBoKDg7m/Pnz0kxJiE7O5RIDgL6DdAyI03HmVC3H0m2THOh0Orp160ZeXp4NInQu+fn5aLVaevXqhaenJ0OGDKGgoEAK1rgIe/ZJaKhPnz5cuXKF4uJiu99LCGE/LpkYAPQb7EXfQV4UFdSSddg2yUF0dDQlJSXWOVtXoCgKJ0+eJCoqCg8PDwDi4+NRqVROt/tEtE1HJQb9+vVDp9ORlZVl93sJIezHZRMDlUrFwCE6evf3ojCvlp+OGtqdHFi2LbrS7oTz589TVVVF3759rc/5+voyYMAAsrOzMRgMDoxO2II9Gyg1pNVqiY+PJy8vT1qVC9GJuWxiAPXJweBhOqL6eFKQU8OJrPb9kuvatSt+fn4u1W0xPz8ftVp9w0ryhIQEjEYjx44dc0xgwiaMRiO1tbUdkhgA3HbbbZjNZnJycjrkfkII23PpxADqk4MhI7zpGePJiawa8rLbnhyoVCr69evH6dOnXWLboqIoFBQU0KtXL7y8GneoDAkJISoqiiNHjmA0Gh0UoWivjtiq2FB4eDjdunXj+PHjsghRiE7K5RMD+FfDpdu8iezlQc4xAwW5bU8O+vXrR01NDefPn7dhhI5RWlpKeXn5TWsWDB8+HL1eT25ubgdHJmylI4obXS8uLo4rV65YSzELIToXt0gMAFRqFcNG+9C9hwfZmQZO5dW06Tp9+vRBpVK5xDqDgoICVCoVMTExTR7v2bMnoaGhHD58WL79dVKWuf6OTAz69euHp6cnx48f77B7CiFsx20SAwC1WkXCGB/CI7Qcy6jm9MnWJwfe3t50797dJdYZFBQUEBkZedNfGiqVioSEBC5fvuwSiZA7csSIgYeHBwMGDCA/P18WrwrRCblVYgCg1qgYMdaXrt20HDlYzdlTta2+RlRUFKWlpdYeEp3R5cuXuXz5crOlj/v164evry8ZGRkdFJmwJUckBlA/nWAymWQRohCdkNslBgAajYrbxvkSEqbl8AE9JWdalxxYVvB35m/RBQUFAPTu3fuWr9NoNAwbNoyzZ89y8eLFjghN2JBer8fDwwOt1vbtyG+la9euhIeHyyJEIToht0wMALRaFaPG+xIcoiFjn57zxS3vIR8aGoqvr2+nnk4oKCggPDwcf3//Zl8bFxeHh4eHlEnuhDqqhkFT4uLiuHz5skss1BXCnbhtYgD17ZpHT/QjMFhD+g9VXDzXsuTA0m2xs25bLC8v5+LFi42KGt2Kl5cXsbGxnDhxgoqKCjtHJ2ypo6oeNqVfv354eHjIIkQhOhm3TgwAPDxUjE70xS9Aw8G0Ki60MDmIjo6mtraWc+fO2TlC27NMI7SmtfKwYcMAOHLkiF1iEvbhyMTA09OTgQMHcuLECVmEKEQn4vaJAYCnp5oxSb74+as5sLuK3OMGFPOt50V79uyJWq3ulNMJBQUFhIaGEhQU1OJzAgIC6NevH8eOHaOmpm1bPUXHc2RiABAbG4vJZJJaGEJ0IpIY/IuXl5pxk/3pEeXBiSwD+3dXUWO4+TSBl5cX3bt373QLEKuqqigpKWnVaIFFQkICdXV10iSnkzCbzRgMhg6retiUsLAwwsLCZBGiEJ2IJAYNaLX1RZCG3ubN5VIju7dXcLn05uWAo6OjuXTpEpWVlR0YZfucPHkSoMXrCxoKCwsjMjKSzMxMTCaTrUMTNuaI4kZNiYuLo6ysTBYhCtFJSGJwHZVKRVQfL8ZP8UOtUfHDrkoKcpruzNgZuy3m5+cTFBREly5d2nR+QkIClZWV5Ofn2zgyYWuOqmFwvf79++Ph4SEjTUJ0EpIY3ERgsJaJU/0Jj/Ag+4iBQ2l66mobTy2EhIR0qm6LBoOB4uJia1nntoiOjiY4OJiMjAwZGnZyzpIYeHp60r9/f06cOCHrU4ToBCQxuAUPTxW3jfNh8DAdF0rq2L29krLSf//D1nDbYmcYWi8sLMRsNrdpfYGFSqVi+PDhlJaWcvbsWRtGJ2zNWRIDqJ9OMBqNsghRiE5AEoNmqFQq+gzQMTbZD7NZYdO6sxQV1Fi/LUdHR1NXV9cpti0WFBTg5+dHeHh4u64zcOBAvL29peCRk7OsMXDk4kOLsLAwunbtKosQhegEJDFooS6hWiZO8yc8QsfRQ9Uc/lGP0ahYty06+zqD2tpaioqK2jWNYKHVaomPj+fUqVOUlZXZKEJha3q9Ho1Gg6enp6NDQaVSERcXx6VLl6S0thBOThKDVvDSqZk6O4IBcTqKi+rY830FNQYNERERTr/OoKioCJPJ1K5phIaGDBmCVquVUQMnZqlh0N5E0Fb69++PVquVSohCODlJDFpJrVbRP1bHmERfamsU9nxfQVBgD8rKypy6XHBBQQHe3t5ERETY5Hre3t4MGjSInJycTt1l0pU5urjR9by8vGQRohCdgCQGbdS1mwcTp/kTGKThyvkwAAoLTzk2qJswGo0UFhbSu3dv1Grb/ZEPHz4cs9nM0aNHbXZNYTt6vd4p1hc0FBcXR11dHSdOnHB0KEKIm5DEoB28fdTcPsmPgbFhaNS+HD5UQFWF8+1OOHPmDHV1dTabRrAICgqiT58+HD16lLq6lnenFB3DkZ0VbyY8PJzQ0FCZThDCiUli0E5qtYrYYT5ER0VRXlXCri1XyT1uwGRynpXXBQUFeHp60rNnT5tfOyEhgZqaGrKzs21+bdF2iqI43VQC1C9CjI2NpbS0VBYhCuGkmk0MLl26xEsvvcR//Md/8Jvf/IbNmzd3RFydzqDY3iiKEe+Ay5zIMvDPrRUt7tRoT2azmZMnTxITE4NGo7H59bt37063bt04fPhwp2xB7aoMhvpqnc6WGED9dldZhCiE82o2MdBoNDz44IO8+eabvPLKK2zbtk0K2zShR48eqNVqrumPET9ahUoFB3ZXcXBvFfoqx/3CLC4uxmAwtKk3QkslJCRQXl5u7cMgHM+Zihtdz8vLi379+pGbm0ttba2jwxFCXKfZxCA4OJjevXsD9SvRIyMjuXz5st0D62w8PT2ZOHEi586dY+v2z4nofY4BcV5cPF/HP7eUk/eTAbMDphfy8/PRarX06tXLbvfo3bs3AQEBZGRk2O0eonUsiYGzLT60kEWIQjgvbWtefPHiRQoLC5v89pmSkkJKSgoAK1euJDQ0tG0BabVtPrcj3Cq+5ORk4uLiWL9+Pdu3b2Pw4MHMuHM6xw8byDlaxbkzJsZM7EpED/t9i2sYn9ls5tSpU/Tv35/u3bvb7Z4AEyZMYNOmTej1+lsmIZ35z9cZtDQ+SyXOyMjIDn0/LY0vJCSE1NRUcnNzSUpKsn9g/+Lsf75COIMWJwYGg4E33niDxYsXNzk8OWXKFKZMmWJ9fOnSpTYFFBoa2uZzO0Jz8anVaubPn8/hw4fZv38/J0+eJCkpiZHjo8k6bGDbxhIienkQO8wbnbft1342jO/cuXNUVFTQs2dPu3+mvXr1QqfT8c033zBv3jx0Ol2z8TkjV4nvwoULANTU1HTo+2nN5zdo0CBSU1PJzs4mLCzMzpHVc+Y/X1vVGBGivVr0m8loNPLGG28wYcIERo8ebe+YOj21Ws2IESNYuHAhgYGBbN26lfTM7xk1UUP/WC/On61j1+ZyCnINmM32m17Iz89HrVYTHR1tt3tYeHh4MGXKFC5dusTXX39tHcoWjqHX61GpVDdN0JzBgAED0Gg00o5ZCCfTbGKgKArvvPMOkZGRzJ49uyNichldunThZz/7GePGjePUqVN8tvYT8DhN4gw/unTVkp1pYPf2CspKjTa/t6IoFBQU0LNnT7y8vGx+/ab07t2b2bNnc+XKFb7++mupiOhAluJGzlIOuSk6nY5+/fqRk5MjdTCEcCLNJga5ubns3r2b48ePs2LFClasWCGLzFrh+tGDbdu2kbp7G7EJcNs4H4x1Cj/srOTwj1XUGGy3e+HSpUuUl5fbvKhRc6Kjo5k7dy4VFRWsW7fOqctEuzJnrGHQFFmEKITzaXaNwcCBA/niiy86IhaXZhk9sKw9KC4uJjExkcQZ/cj/qYaC3BouFBvpF+tFVB8vtNr2fdMrKChApVJZd5R0pJ49ezJv3jw2btzIunXrmD9/PoGBgR0ehztzxqqHTenevTvBwcEcP36c2NhYR4cjhEAqH3aohqMHQUFBbNu2jW3bNtOrj5nE6f4EhWjIzjSwc1M5J3MNmIxtX3+Qn59PRESEw345dO/enbvuuova2lq++uorrly54pA43FVnGTGwtGO+cOGC0y4KFMLdSGLgAF26dOHuu+9m/PjxFBUV8cknn1ByLp/RE30Zm+yHf4CGrEwDOzaVc/JETasThMuXL3P58mW7FjVqibCwMO666y7MZjPr1q2jrKzMofHYm9lsdorqj5ZyyM5aw+B6AwcORKPRSCVEIZyEJAYOolarSUhIaDR6sGnTJjx11dw+yY+xk/zwC9CQdbi61QlCQUEBgEOmEa4XGhrKggULUKlUrFu3jpKSEkeHZBeKorBx40bWr1/v8OSgtrYWk8nUKUYMoL4IU9++fWURohBOQhIDB7t+9GDNmjVs2rSJKkMJtyf5cvt1CULhiZpmGzQVFBQQHh6Ov79/B72LW+vSpQsLFizAw8ODNWvWWIvvuJK8vDzOnDlDcXGxw7/5VldXA85ZDvlm4uLiqK2tJT8/39GhCOH2JDFwApbRgwcffJCEhASKi4vZsGEDH3/8MafPHiXhdi23T/LF11/N8cPV7NxUTmFe0wnC1atXuXjxosOnEa4XFBTE3XffjY+PDxs2bKC4uNjRIdmM0WgkLS2N0NBQevbsyQ8//ODQrZrO3CfhZiIiIggODubYsWOODkUItyeJgRMJCAhg3LhxPPzww0ybNg1vb2/27t3LBx98wOEj/6RvrJ7bk3zx8VNzPKPpBMHS/rijtym2hL+/Pw8//DB+fn5s3LiR06dPOzokm8jMzKSiooIJEyYwadIkTCYTu3fvdlg8zt4noSmWdsznz593+bUoQjg7SQyckFarZeDAgfzsZz9j4cKFDBo0iPz8fD7//HNSdq0jKPw0I8d74uP77wTh1L8ShJ9++omQkBCCgoIc/TaaFBAQwIIFCwgKCuKbb76hsLDQ0SG1i16v5+DBg8TExNCzZ0+CgoIYOXIkeXl5nDp1ymExQecaMYD6RYhqtVoqIQrhYJIYOLmuXbuSnJzMww8/TGJiIkajkR07drDxu48xeRxmcEIt3r5qjmVUs/2bCxQVFRET43yjBQ35+Phw1113ERoayqZNm8jLy3N0SG22f/9+TCYT48ePtz6XkJBAcHAw//znPx2ymK4zjhhA/c9Fnz59yMnJwWi0fTVQIUTLSGLQSXh5eREfH8/999/PggUL6NWrF0ePHmXTlrVcvPY9kX3OY6irH5q/UBTO0UN6yq+aHBz1zel0OubPn094eDhbt24lJyfH0SG1WllZGVlZWQwZMoTg4GDr81qtlkmTJlFeXs7Bgwc7PK7q6mp0Oh0ajabD791esbGxGAwG684aIUTHa1XbZeF4KpWKyMhIIiMjqaqqIjs7m2PHjrH37HYAAgODie4dxplTtRQV1NKlq4aYvl506+GBWu1cdfO9vLy48847+e6779i+fTtGo5G4uDhHh9Vie/bswdPTk1GjRt1wrEePHgwaNIiMjAwGDBhASEhIh8XVWYobNaVnz54EBASQlZXFgAEDHB2OEG5JRgw6MV9fX0aOHMnixYuZPXs257hm5wAAIABJREFUvXv3Jjk5ieGjfZk6J4DB8ToMeoX0fXp2fFfOiSwDhmrHF+BpyNPTk7lz5xIVFcXOnTtJT09HUezXcdJWTp06xenTpxk1atRNh+zHjx+Pp6cnu3bt6tD31JmKG13Psgjx7NmzXL161dHhCOGWJDFwAWq12trZcPjw4QB4eqnpM1BH8h3+jJrgi3+ghtzjBlK+LSd9XxVlpUan+QWs1WqZNWsWffv2JS0tjc2bN1NTU+PosG7KbDazZ88eAgMDGTp06E1f5+3tzfjx4ykpKbHuFukInXnEAGDw4MGoVCpZhCiEg0hi4OJUahXhER6MSfRj0h3+xPTz4uK5On7YWcnu7RUUFdRgbEdPBlvRarXMnDmT8ePHc/LkSdauXcvFixcdHVaTjh8/zpUrVxg/fnyz8/iDBg0iIiKCvXv3WhcF2ltnaaB0M76+vsTExPDTTz9hMjnvOhkhXJUkBm7Ez19D7HBvps4NZOht3qDA0UPVpHxTTtbhaqoqHPuPsEqlIiEhgQULFmA0Gvnyyy85fvy404xsANTU1LB//34iIyNbVHJapVKRnJxMXV0de/futXt8RqOR2traTp0YQP0iRL1e3+m3swrRGUli4Ia0WhVRfbyYON2fscl+dO2mpTCvhp2bK0jbWUFBjoHKcsclCRERESxcuJDIyEh27tzJ9u3bqa2tdVg8DR08eBCDwcCECRNQqVq2mLNLly6MGDGCnJwczpw5Y9f4OutWxetFRUXh5+cn0wlCOIAkBm5MpVIR0lXLiLG+TJkTwIA4HcZahewjBnZtqWDnpvqRhEsX6zCbO/Zbu4+PD3PnzmXMmDHk5uby+eefO7wi3rVr18jMzGTQoEGEhYW16tyRI0cSGBjIrl277LpHv7MWN7qeWq1m8ODBFBUVUV5e7uhwhHArkhgIAHTeavrH6kicEcDk2QHEJXjj46fmVH4N+3ZVsX1D/aLFs0W11NZ2zM4GtVrNqFGjmD9/PgaDgc8//9yh9Q7S0tJQq9XcfvvtrT5Xq9WSlJTE1atXSU9Pt0N09VwlMYD6RYhAhy7cFEJIHQPRBB9fNTH9vIjp54WxTqH0Qh0XSoxcKKmj5HQdKhV0CdUQHuFBeKQHfv72LaTTs2dPFi5cyNatW9m+fTslJSVMnDgRrbbjfnxLSkrIz89n9OjR+Pn5tekaUVFR9O/fn4MHD9K/f/9GRZFspTN2VryZgIAAevXqRXZ2NqNGjUKtlu8xQnQE+ZsmbknroaJ7D0+GjfJh2p0BjJ/sR5+BXtRaphw2V7BzczlZmfadcvDz8+Ouu+5ixIgRHD9+nC+++KLD9rkrisKePXvw9fUlISGhXdeaMGECWq3WbrUNXGnEAOrbMVdWVlJUVOToUIRwG5IYiBZTqVQEh2oZNNSbpBkBTJ7tXz/l4KvmVN6/phw2lnP4xyrOF9dhsvE2SLVazbhx45gzZw4VFRWsXbuW/Px8m96jKbm5uVy4cIGxY8fi4eHRrmv5+voybtw4zp49a5dpEb1ej4eHR4eOpthTTEwM3t7esghRiA4kiYFoMx9fDTH9vBiT6Mf0eYHcNs6H8O5aLhQbObi3im0brnEwrYqzp2qps+G6hJiYGBYuXEhwcDCbN29m9+7ddtvvXldXxw8//EBY2P9v796joyrvRo9/99xnMpOZ3MiVkBBClHARDIIgF8FSxQuUY/HyWk+7fJe+C3us9dVjPWetrq6lbbXqq+0rnvq6rMf6Hq21Bam2InIXRe4XCSC3EAi5kdvcr3s/549JJgkkEEgyCbzPZ63J3jN7z+Q3e/bM/u3ntkdwzTXXDMhrjh8/npycHLZs2UIoFBqQ1+xwpQ9udC69Xs+4ceOoqqrC7/cPdTiS9F+CTAykAdFR5TB5egoLFqcyfU4KBUUmWpti7NkW4LOPPHy9ycfhA+4BGZY5NTWVu+++m0mTJrF3717+8pe/4PV6B+CddLdnzx58Pt8ldU+8mI6xDUKhEF9++eWAvGaHK31wo56Ul5cjhJCNECUpSWRiIA04nU4hK8fIxAob37kr3i5hdJmZgE9j66azfP43D1vWejl2ONSvQZX0ej1z5szhtttuo6Wlhffff58jR44MWN293+9n165dlJSUkJ+fPyCv2SEzM5PJkydTWVlJbW3tgL3u1VZiAOByucjPz6eysnJYDXYlSVcrmRhIg6qjXcK4SVZuXuhg8b0jKRtvQVXh0L4Q6//hZeNqD4e/CeJuvbzrN5SWlnLffffhcDhYvXo17733HkeOHEHT+lcysXXrVlRVZebMmf16nd5MmzYNh8PB+vXrB6wq5GpMDCBe/eLxeAZ9gChJkmR3RSmJFEUhLcPM2HILY8stBPwq9TVR6s5EOXoozNGDYYwmhVSXnlSnrn2qx+7UYzBcuBjf5XJxzz33cPToUXbs2MHq1atxuVxMnTqVsrKyS+7qdvbsWQ4ePMjkyZNxuVz9edu9MhqNzJ07l48//pg9e/ZQUVHRr9dTVZVQKHTFj3rYk5KSEiwWC5WVlRQWFg51OJJ0VZOJgTRkbCl6RpfpGV1mIRzSaKiN0tqs4mlTOVUVQe0yQGCKvT1RcOlxtCcNthRdt3p/nU5HWVkZY8eO5fjx42zfvp3PP/+cbdu2UVFRwbXXXnvRix5BZ/dEi8XCDTfcMBhvPaG4uJiSkhK2b99OaWkpTqfzsl+royHj1VhiYDAYuOaaa9i/f/9VWyoiScOFTAykYcFs0VE42kxh+3WJhBAE/BqeNhVPm4bXHU8Y6mqiiefoDZDq1ONw6jtLF1J1mMwKY8aMoaSkhKqqKnbs2MH69evZvn07119/PeXl5RfszldVVUVNTQ1z5szBbDYP9ltn9uzZ/Od//icbN27krrvuuuxGjlfbGAbnKi8vZ+/evRw+fLjf40lIktS7iyYGr7/+Ort378bpdPLyyy8nIyZJQlEUUux6Uux6cgs6H4/FBF63mkgUPG6Nupoop050XmTJYKT9uTps9lymVyzC46vl4OFdbNq0iR07djBlyhQmTJhw3rgEsViMLVu2kJaWxvjx45PyXh0OBzfeeCObN2/mwIEDXHvttZc1DsHVnhhkZGSQk5NDZWUlkydPHrBeIpIkdXfRX5+5c+dy6623snz58mTEI0kXZDAopGUYSMvo3HWFEIRDAk+bis+r4feq+H0a7tZ4CUO8PWM6VnELeRkNuAP72bJlC9u27WTM6ImUl0/AlWbBatWxc+dO2trauPPOO/tU7TBQJk6cyLfffsuGDRv44osvyM3NJT8/n4KCArKzs/sUy9WeGEC8EeLatWupq6sjLy9vqMORpKvSRRODcePG0djYmIxYJOmyKIqCxapgseoYkdt9maYJggENv08j4NPwe0fh9xXQeLaOurP7OPTtdr49spdU27U4bKOpbV6L05FHyJvNmVMRHKl6Uhw69PrBPTvV6XQsWbKEU6dOcebMGWpqavj666+BeP16bm4uBQUFlJeXYzabe0wUOq6TcDU2PuxQWlqaKFmRiYEkDY4Ba2Owdu1a1q5dC8Dzzz9PZmbm5QVkMFz2c5NBxtc/wye+IoSYzrFjp9i8eRPV1fto8+8DFLIzpnH8cAQh4tUTigKOVCOudBOuNBOudBPOdCMulwmDcWB7/ObmdmY2gUCA6upqqqqqqKqqYuvWrWzduhWj0UhhYSHFxcUUFRWRn5+PXq9HCIHBYCAvL2/IitmT8fl2DGqVkpJyyUnQxeLTNO2yL9bkDcc43uTnuvzLb0AqScOBIvrQcbyxsZEXXnjhktoYXO6gLZmZmTQ1NV3Wc5NhMOILxzRqPBFOtYUJxTQMOgWDTkGvUzDqFPQ6Eo+de9PrFAxdlo/Oz6a5uXlA4xtIw/XzPXq6ji1f7yA3L49bZ1agqgK/V8PrUfF5VLzu+Lzfq9H1G2NL0WFP1eFIjTd8THHosVgUzFbdRbtYXqpgMIjX6+XQoUPU1NQkPmej0Uhubi5+v59IJMKPfvSjAf2/lyIZn29jYyN/+tOfmDNnDpMmTQIgFNN49atajjWHWDDGxYIxLlzW8897eosvHA6zevVq2traWLx48SX1DqnxhPnkcCsbqtwYdQp/WDIGk/7SkwtZAiINF7JXQhKpmqDOF6G6LcyptjDVbfH5el+EgbooYXlOAz+dPoKslP5d7GcwCFVFhIIIvw9iUYjF2qdd5oUAvR50etDpOqd6XffH9Ocs1+nj6yi6Pp8tN/qibD3tZetpL4fPBhGMhZNQY27gv08ekege2ZWmCfy+eC8Jn6c9cXCrNDXEOHc8JYMRLBYdFqsOc3tVh9kSn1qsuktOIKxWKyNHjmTEiBEIIQh6PZw5fZoztbWcqaujuc1NwYgsxJlq0LT2m9rLfPf7omO+I+vpug0Vpct9pcusAigdDwMKodRUhMcNif1ZJF4zPmlfIDof77IQFAWl4/PXG+Kfqd7QuU/oDWTp9WS5nFTu28uEnCzcqp7ndns54Y4xNs3I/9vfxAffNDEj28DCPD1lNg1FU0GNEalLQbS0xPc3NYpQVTz+AJ8cOUlbKIxBp+Ov7/0/vjfhGlxOJ5jMnTdz57xmNLKvIcTHh1vZXefHoFOYXeTgjrL0y0oKJGk4kYkB8cZriR9KRQG9oV9FsUIImgKx9oN/mGp3fFrjjhBtzwB0CuTYjRQ6TcwqsDLKrqcwRSFFpxGLqcSiMWIxFVVV4/djKjFVQ42pRFUNVdWIddw0QUzV8MYUVjRqPL6qjccdtVxv8nYeBFQVRB/mNa3bfSF6PpD0el9Vzznod5kKjaS1Vuk4mCm6+EFL0YGiUGvN4OuMcramj+O4PT7McVGgnntbD3OD+yjrM6/jY6ZyZNd+njyxkoyIp+NTTRzTbEJgA7K7PKahELRkEjBnELakETa5CJldhE1OwkYnLaZUwgYHmu78hM2ghTGrPiyaH2vMjT18Fnu4AYe/DkuoGSUWSSRQjTEVEY2AGsMMjG6/AQT1RvSVGtq69wdvu9LlmN8D9yC/fodx6flsyr+W/S//iuVj76fNZOd/HnyPG5oPUmvN5NP8G9kQrWBzvZVi7xlurd3K7IY9aFq02+s0WFP5+6hJqDodd1Tvx6pGWVU8hRXbdrLoxG7SIoFu6wf1JjZmX88/CmZyxjYCV8TLvS37WOA5hOuYCjvS4bGfD8BWkKShc9GqhFdffZWDBw/i9XpxOp0sXbqUefPmXfSFB7oqQWga+H3gc4PXDT4PwutJzOP1IHxuiMR/NOMHtlj8QKWq5z/WfgaB2v7YuXQ6MBg6z1b0RtDr0ZtMqCjdlxkM+A1WPrKXU2kYwSmDk4DOlHip9KiXUaEmCoONFPrrKfTWUeCrwxwNdT9rGgB11gxeLP8BJ+15LKlez30n16AXXU5l9fr4QVKn6z6fuJ17Jt7Tsh7ud7yOwYBiMMa3j8HYfjMkpilOF/5wuNsypet6KPHEpOMz0jRER/Kidk1G1M5EpuOxjvUEianQNE5rFraqaWzV0qkWKQCU4uFGpZnpopFcgkA8ObRYrKwLpfKacg0mNP5VqWRix+FOSfzpPIM+98xaiPbYtB7fRwQjYWEhpFgJK1bCSgohnY2wLoWQ3k7A4CJisHd+XCKKXWvDgQc7XtKtUcxaG1ZDFMVgAKOxy7aM76PKeaUpPX1+vcwruvZ9suPMHrqf5XdE1vWMv/Nxl8tJW5u7h23VpWShS+kDic3Xnsh1fM7nfU/VzpumEgmHeXPHN9SbcznluIb/ndlAqT4Qj8NgAIOBoGJgs9/Op24L1WEDKXrBwnwj89Ii5KYYOd54ljU792CzmLnz5jmkp2cAgubGRlZu2IQCfG/KRNJNRur9ET5tNvG510ZA6Bmj83O7rpaZsVqM0RBEIohwCMVsQffwU334pp5PViVIw0Wf2hhcjktNDIQQiPWfYFNjBBrr4gd9nxu8nviB3+eN/2j0xGIFhxPsqfGivsTBPH5TEsWS+i43Q2eRdMdNp4//2CV+iGLtRY6d82ajgbDfHz9YqTGEGmOzqZB3HFNw66xcE25glNpGoephlPAxEj8OA/H/ZzTGpx1JRdf7hq7L4vEp+q5xd5kmkpbzl2dkZlHb3MpbBzysOemnPNPCv87IIT3FhHKZjaoGUjLqoIUQnGgN89WpeDXBGU8EBbg2y8qNhQ5uHOnotaqlI77T7jDPbz5DrTfCP03KYsm4dHRJatAXCWt4PVp72wY1MR8Kdn5VdXoS7RocqfFBnuypOmwpOnS6oevfn6w2JJtPelj92RqyIg18/4EfUpCW0uu6QggOng3y929b+brGh6pq3Giqw1Z/gOzsbO68887zuni2tLSwYsUKoqqGe9RMtrXEq5RmFDq4oyyNazKtA97AUyYG0nAxbBIDAPXH90AkBCmO9gN9fKrYneBITRz8FUcq2J2d943Jq0/v+sN3sjXEGzsaOHg2SGmGhUemZlOaMbRdxbrGt7HKzevb6rEYdfzrzDwm5fT+45ksg3XgCERVjjWH2FXrZ+tpLw2+KDoFxo+wMaPQwbSRDtJ7aIx2ofiCUY3l2+r4otrLDQV2fnJjLnZT8sY2OFcopBFVbZw61YLXreH3qYR9AjXcfb3Odgyd7RnOfcxkVgal58JgJwZCCFYcbOGPe88yKSVI5qkvmDdvXp8Ho1JNKfzhvQ8J1h6jwZRNU/ZkvluWwS0lLlLN8c82ompsPunhs29Ok3X6K/SKRvp187lzSjGZtsH7rZGJgTRcDKvEYPvxRm64thgi/kGIaGBkZmZSXdvA+/ub+PuRVlJMeh68LotbSpxJO6O8WHxdf5hPucO8sPkMZzwR7p2YyffLM9Bf4WeUUVVwsi3E0eaOW5AadyReiqyDSTkp3DjSwbQCO6mWS2tGc258Qgj+fqSVP+xqJCvFyNOz8hmdbulX/D1pCcZYd7yNqtYwwahGKKYRjGkEo+23mEZE7fmrakTBhQGXYsCh6BlhMjAqxYINHeGQIBI+/3mKDiztyYK5vSFkR+JgsrQ3jLTEE4hLKYEYzMRA1QT/sbOB1UfbmDXKwf+YnsOHf3ofo9HIPffcc9Hnh8Nh1q1bx7Fjx5g85XooGM+nR9uobAxi0ivcNCqVdKuBNcfa8IRVRjnNLBipp2HnWlQ1xuLFixkxYsSgvDeQiYE0fAybxCAY1Xjwr0dRNcGUvBTmjXYyNd+OcRi18BVCsKtJ8O+bj+MOqXy31MUDk7JwmIfuLPJcPf0wh2Ia/2d7PRurPFyXY+OnM/NwXeIBczDjuxBNCGo9kUQCcKQ5RFVrmFh7I06nWU9phoXSDCulGRbKsqz9OqvvLb7DZ4P85oszeCMqj0zN5paS/l9xUROC/fUBVh9tZVuND01AnsOIzajHatTFb4bu0wynAy0S7Hz8nHX21vv5a2Uzp9wRcuxGvjcunZuLUlEjCuGgRiikEQoKQkGt/RafDwcF0WjPPwUms4LZHE8UzJaepp1JxIgRWYOSGASjGi9tOcPOWj9LxqXzg+uy0CkKe/bs4YsvvuD++++/4PgEXq+Xv/3tb7S2tjJ37txuJQwnW0P840gbm066CccEUwvs3FmWxoRsG4qi4Ha7WbFiBZFIhMWLF5OdnT3g7w9kYiANH8MmMQCocYfZWh/lH5X1tARjOEw6ZhelMr/Exeg085COjT4cqw160mvjTSH4/Lib/9jRgMOs58mb8igfkfyhcy+UGAghaA7G4klAU5CjzSGOtYQIRONtSywGhTHpnUlAaYaVrJT+9SC5lPjcoRgvfVnL/voA3ylx8vDU7MvqmtYWirH+uJvPjrVR74uSatYzf7ST75a6yHWYLvjcviRWmhDsqPHxYWUzR5tDpFkNfO/adBaMcWG9wIBMsZggHNIIh7pOz3ksHJ/21F4X4hfDMpriU7NZiScV7UmD2aJgNuswtU8NRvr02bUGYzy78TRVrWEersjmtrFpiWXBYJC33nqL8ePHM3fu3B6f39jYyMcff0w0GuXee+/t9TLagahKOCZI66HKyePxsGLFCkKhEIsWLeo2ENVAkYmBNFwMq8QA4j98DY1n2VfvZ90JN9tO+4hqglEuM/NHO5lTlNrjwCWDxRdReX9/E/9orzZ49KZipmXrh0W1QU8uduA40RLiN1vO0OCL8oNJWSxOYqO63uKr9UTYeNLNpioP9b54dzK9AkVplvYEwMLYDCv5qaZBrwa52PZTNcH7+5v4sLKZknQzT8/KJ9t+4YM5xJOeA40BPjvaxtbTXmIalI+wcmtpGjeO7HvJ2KWUuAgh2N8Q4C8HmtnfEMBh0nHHNencPjat36VcPScRGgpmPO4g4bCWqMaIRnr+idHp6J44mBWMJgVFpyQ6SXgiKutPuAnENG4ucVKcbo53nlAUdPp4R4ZtO9ZSV3+K7y36IVarAWuKDqMx3obixIkTrF69GqvVyl133UVZWdlll2h4vV5WrFhBIBBg0aJFA34gl4mBNFwMy8Sg6xfXF1b5otrD+hNujjSH0ClwfZ6d+aOdVOTbMQ7SGPZCCDZUefi/exrxhFRuLXXxT5OyKM7PHpYj93Xoy4EjEFX596/r+eqUl6n58UZ1yaoO6YjPE4rxRbWXjVXxz1UBJuTYuCHfzthMK8Vp5iEZKKavB94dNT5e2Rrfx5+YkUdFvr3H9bzh+IHts2NtnPFESDHpmFccLx0Y6bz0Szpfbh3+t01B/lLZzPYaHxaDjttKXdx1bXqfGmT2Nz5NFUQinSUOkZAgHNbap/HHI+3TaFTEh9EQvXdCOlcwUk996xoyU2fisJYAoDcIApFvqT27HYc9ixsqvktamp28/HQiUU88AbmMhNjn87FixQr8fj933nknBQUFF39SH8nEQBouhn1i0NVpd5j1J9xsqPLQGozhMOuZU5TK/NFOigewqqGqvdrg0NkgYzMsPDI1hzEZlovGNxz0Nb6ORnVv724k3WrgqZvyGZs5uFUj4ZjGYY/Cx/vPsLvWhyqgyGVmTnEqc4pSyRjEFt99dSmfb703wgtfnOFEa5jvl2dw38RM9DoFIQSHm4KsPtrGl9VeopqgLNPKraUuZhY6MBsuP+Hp7/53sjXEXw+2sKXag15RuKXEyffGpfep1CMZ8XXYVOXmd1/XkZti4n/NzifLZkyMoyU0gdY+XIQQAlUVrPrbe1isNubctAi/L8Y3B76ipu4gTnshmY6b0LTuCZDeADabDmtKvItnx9RkUuJDUbQnJkKI+PANiWRFEAwG+PLrjwkEfUyd8l0y0vITAznq9DDmmstrnCoTA2m4uKISgw6qJthb117VUOMjpgmKXGbmjXYyymWOj6KqKOgUpcs86HQKekVBr9Djspgq+HNlM58eacXe3ttg/jm9Da6WxKDDkaYgL245Q0swxo+mjOD2sWkDWmevCcGBhgCbTnr46pSXQFQj3WpgTlEqc4tTKUob+Bb+/XGp2y+iavzHjgY+P+5mYo6NaQV21hx1U+0OYzXomFucyq2lrgF7nwO1/9V5I6w82MK6E240IZhdlMp/K8+g8DJKMQYyPiEEf61s4d19Zxk/wsozswuw96E0a+fOnXz11Vfcc889bNu2jZMnTzJlyhRmzpwJQDQqCPo1DHo7DXVtBPwawYCIT/1arw0ve6NqQepaPycW8zLCNRebOT6KptmisGDR5V1ESSYG0nBxRSYGXXnDKluqPaw74eZoc+iy/mdXCiSqDXoqXr/aEgOIb8Pfbq1lxxk/WTYDWSlGMmwGMmxGMm2GxHyGzUCaxdCnev5TbWE2VLnZdNJDcyCGxaBjRqGDRZNGMtISHdIukxdyuZ/v2uNtvLGjgYgqKEm3cGupi1mjUi/Y2C+Z8fWmORBl1aEWVh9tI6wKJufGu3reUGDvsRHeYManaoLf76hnzTE3s4tSeWx6Tp/bXgQCAf7whz+gKAqapjF37lwmTJjQ5/iikfjluSMRDZ2idIygHW/L0DHf3rZBUeLtH0KhIJ/8fRVtba3cdutCikcX9SuplomBNFxc8YlBt//pidAaiqF2FDUKER9dVYhz5juXaUKgdpmflJNywX7qV2NiAPH3vuZYGwcbgzQHojQFYjQHYolrO3TQKZBmMZBuM7QnDe1JhDX+2ImWeEJQ1RpGp8Dk3BTmFjuZVmDHbNBdtdsPoMEXIRjVBrUUZLC2nyes8vdvW9jYpQFoWaaFGwocTC+wU9DHkoRLja81GGN/vZ/9DQH21/tp9Me4uzyDf5qUecmNYlevXk1VVRULFy5k1KhRAxLfxQSDQT766COam5tZuHAho0ePvviTeiETA2m4uKoSg2T4rxSfEAJvRKM5EKU5EKOpfRq/dSYPwVj3VmKlGRbmFqdy06jU88ZL+K+0/QZDMkYWPOWOsK3Gy7bTPo61xEvh8hwmpo+0c0OBnbJMa68H7YvF54uoHGgIJBKB0+4IACkmHROybcwuSmVmYeplxR6LxYjFYlgsyU3sQ6EQq1at4uzZs9x2222UlJRc1uvIxEAaLuTVFaVeKYpCqllPqllPcVrv6wWiaiJhyEwxUJDav3pqaegoisIol5lRLjNLx2fSFIiyvcbHttNeVh1qYcXBFlwWPVPz7Uwf6WBiju2CvUfCMY1DZ4Psq/ezvz7AidYQmgCzXmHcCBvzip1MzEmhOM3c7+olg8GAwZD8nzSLxcLixYtZtWoV69evZ+TIkZhMA9OYU5KGgkwMpH6zGfXYnPrL6n4nDW+ZNiMLx6axcGwavojK7lo/22q8bKn28vlxNxaDwuRcO9MK7FTk23GpGocaA+xrCPBNvZ/DTSFimsCgg7EZVpaOz2BiTgpjM6yD1tV4KJjNZhYtWoTX65VJgXTFk4mBJEl9YjfpmV2UyuyiVKKqxjcNAbbV+Nhe42PraS86BcyG4wSjGgowOt3MnWVpTMyxMW6EDUs/umleCcxmM2azTI6lK59MDCRJumRGvY4peXam5Nl5ZKrgWHMJ1Q3QAAAK8klEQVSIbTU+MJgYk6pjfLZtWF1DRJKkvpOJgSRJ/aJTFMZmWhmbaR32jTclSbq4q7tsT5IkSZKkSyITA0mSJEmSEmRiIEmSJElSgkwMJEmSJElKkImBJEmSJEkJMjGQJEmSJClBJgaSJEmSJCXIxECSJEmSpIRBu7qiJEmSJElXnmFXYvCzn/1sqEO4IBlf/8j4+kfG1z/DPT5JGg6GXWIgSZIkSdLQkYmBJEmSJEkJ+l/84he/GOogzjV69OihDuGCZHz9I+PrHxlf/wz3+CRpqMnGh5IkSZIkJciqBEmSJEmSEmRiIEmSJElSgmGo/vHevXt5++230TSN+fPns3jx4m7Lo9Eor732GidOnMDhcPD4448zYsSIpMTW1NTE8uXLaWtrQ1EUbrnlFhYuXNhtncrKSn7zm98kYpo2bRp33313UuIDePTRR7FYLOh0OvR6Pc8//3y35UII3n77bfbs2YPZbGbZsmVJq1utra3llVdeSdxvbGxk6dKl3H777YnHkr39Xn/9dXbv3o3T6eTll18GwOfz8corr3D27FmysrL46U9/it1uP++5GzduZMWKFQAsWbKEuXPnJiW+d999l127dmEwGMjOzmbZsmWkpKSc99yL7QuDFd+f//xn1q1bR2pqKgD33XcfU6ZMOe+5F/uuD1Z8r7zyCrW1tQAEAgFsNhsvvvjiec9NxvaTpCuKGAKqqoof//jHor6+XkSjUfHkk0+K06dPd1tn9erV4o033hBCCLFlyxbxb//2b0mLr6WlRRw/flwIIUQgEBCPPfbYefEdOHBA/PrXv05aTOdatmyZcLvdvS7ftWuX+OUvfyk0TRPffvuteOaZZ5IYXSdVVcU///M/i8bGxm6PJ3v7VVZWiuPHj4snnngi8di7774rVq5cKYQQYuXKleLdd98973ler1c8+uijwuv1dptPRnx79+4VsVgsEWtP8Qlx8X1hsOL74IMPxKpVqy74vL581wcrvq7eeecd8eGHH/a4LBnbT5KuJENSlXDs2DFycnLIzs7GYDAwY8YMduzY0W2dnTt3Js7Mpk+fzoEDBxBJaieZlpaWOLu2Wq3k5+fT0tKSlP89UHbu3Mns2bNRFIWxY8fi9/tpbW1NehzffPMNOTk5ZGVlJf1/dzVu3LjzSgN27NjBnDlzAJgzZ855+yDEz3YnTpyI3W7HbrczceJE9u7dm5T4Jk2ahF6vB2Ds2LFDug/2FF9f9OW7PtjxCSHYunUrM2fOHPD/K0lXoyGpSmhpaSEjIyNxPyMjg6NHj/a6jl6vx2az4fV6E8WWydLY2EhVVRVjxow5b9mRI0d46qmnSEtL4wc/+AEjR45Mamy//OUvAfjOd77DLbfc0m1ZS0sLmZmZifsZGRm0tLSQlpaW1Bi//PLLXn+Qh3r7ud3uxPZwuVy43e7z1jl3X01PTx+SA/T69euZMWNGr8svtC8Mps8++4zNmzczevRoHnzwwfMOzn35rg+2Q4cO4XQ6yc3N7XWdodp+kjQcDVkbgytBKBTi5Zdf5oc//CE2m63bsuLiYl5//XUsFgu7d+/mxRdf5He/+13SYnv22WdJT0/H7Xbz3HPPkZeXx7hx45L2//siFouxa9cu7r///vOWDfX2O5eiKCiKMmT//0JWrFiBXq9n1qxZPS4fqn1hwYIFiXYhH3zwAX/84x9ZtmzZoP/fS3Wh5BSujO+SJCXTkFQlpKen09zcnLjf3NxMenp6r+uoqkogEMDhcCQtxlgsxssvv8ysWbOYNm3aecttNhsWiwWAKVOmoKoqHo8nafF1bC+n08nUqVM5duzYecubmpoS93vaxoNtz549FBcX43K5zls21NsP4tuuo3qltbW1x9Koc/fVlpaWpG7HjRs3smvXLh577LFeE5eL7QuDxeVyodPp0Ol0zJ8/n+PHj/cY28W+64NJVVW2b99+wdKWodp+kjRcDUliUFJSQl1dHY2NjcRiMb766isqKiq6rXP99dezceNGAL7++mvKy8uTdkYnhOD3v/89+fn53HHHHT2u09bWlmjzcOzYMTRNS1riEgqFCAaDifn9+/dTWFjYbZ2Kigo2b96MEIIjR45gs9mGVTXCUG6/DhUVFWzatAmATZs2MXXq1PPWue6669i3bx8+nw+fz8e+ffu47rrrkhLf3r17WbVqFU8//TRms7nHdfqyLwyWrm1Wtm/f3mNVUF++64Ppm2++IS8vr1t1RldDuf0kabgaspEPd+/ezTvvvIOmadx8880sWbKEDz74gJKSEioqKohEIrz22mtUVVVht9t5/PHHyc7OTkpshw8f5uc//zmFhYWJZOS+++5LnIEvWLCA1atXs2bNGvR6PSaTiQcffJCysrKkxNfQ0MBLL70ExM+IbrrpJpYsWcKaNWsS8QkheOutt9i3bx8mk4lly5ZRUlKSlPgg/iO7bNkyXnvttUQ1TNf4kr39Xn31VQ4ePIjX68XpdLJ06VKmTp3KK6+8QlNTU7fuisePH+fzzz/nX/7lX4B4/f7KlSuBeHfFm2++OSnxrVy5klgslqi3Ly0t5eGHH6alpYU33niDZ555ptd9IRnxVVZWcvLkSRRFISsri4cffpi0tLRu8UHP3/VkxDdv3jyWL19OaWkpCxYsSKw7FNtPkq4kckhkSZIkSZIS5MiHkiRJkiQlyMRAkiRJkqQEmRhIkiRJkpQgEwNJkiRJkhJkYiBJkiRJUoJMDCSpF0uXLqW+vn6ow5AkSUoqOSSydMV49NFHaWtrQ6frzGfnzp3LQw89NIRRSZIkXV1kYiBdUZ5++mkmTpw41GFIkiRdtWRiIF3xNm7cyLp16ygqKmLz5s2kpaXx0EMPMWHCBCA+0t2bb77J4cOHsdvtLFq0KHEFPU3T+Oijj9iwYQNut5vc3FyeeuqpxJUp9+/fz69+9Ss8Hg833XQTDz300LC92JIkSdJAkImBdFU4evQo06ZN46233mL79u289NJLLF++HLvdzm9/+1tGjhzJG2+8QW1tLc8++yw5OTmMHz+eTz75hC+//JJnnnmG3Nxcqquru12XYPfu3fz6178mGAzy9NNPU1FRkbRrJUiSJA0FmRhIV5QXX3wRvV6fuP/AAw9gMBhwOp3cfvvtKIrCjBkz+Pjjj9m9ezfjxo3j8OHD/OxnP8NkMlFUVMT8+fPZtGkT48ePZ926dTzwwAPk5eUBUFRU1O3/LV68mJSUFFJSUigvL+fkyZMyMZAk6aomEwPpivLUU0+d18Zg48aNpKendyviz8rKoqWlhdbWVux2O1arNbEsMzMzcYng5ubmC16cq+slo81mM6FQaKDeiiRJ0rAkuytKV4WWlha6Xg+sqamJ9PR00tLS8Pl8iUvrdl0GkJGRQUNDQ9LjlSRJGq5kYiBdFdxuN59++imxWIytW7dy5swZJk+eTGZmJmVlZbz33ntEIhGqq6vZsGEDs2bNAmD+/Pl88MEH1NXVIYSguroar9c7xO9GkiRp6MiqBOmK8sILL3Qbx2DixIlMnTqV0tJS6urqeOihh3C5XDzxxBM4HA4AfvKTn/Dmm2/yyCOPYLfb+f73v5+ojrjjjjuIRqM899xzeL1e8vPzefLJJ4fkvUmSJA0Hiuha/ipJV6CO7orPPvvsUIciSZJ0xZNVCZIkSZIkJcjEQJIkSZKkBFmVIEmSJElSgiwxkCRJkiQpQSYGkiRJkiQlyMRAkiRJkqQEmRhIkiRJkpQgEwNJkiRJkhL+P+NtgWy42COdAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZKaNt1HAvsW"
      },
      "source": [
        "###Performance\n",
        "\n",
        "Your final hand-in accuracy should be at least 0.75. If it is currently not, try modifying your head architecture. You will also add some additional regularization methods in the conceptual questions below which may help."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZXhTIP9KaExe"
      },
      "source": [
        "##Conceptual Questions\n",
        "\n",
        "Please answer these questions in the context of this mini-project. For example, there is no issue with the data at all. :-)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTd2jf1f96mu"
      },
      "source": [
        "1.\n",
        "\n",
        "The base ResNet architecture includes batch normalization, so please also include this in your head model.\n",
        "\n",
        "a. It is a common practice to freeze the batch norm parameters when fine-tuning. Why is this?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_B82518XaH7Z"
      },
      "source": [
        "2.\n",
        "\n",
        "Imagine you plot your training and validation losses per epoch and notice that while your training loss converges to 0.05 at epoch 40, your validation loss hits its minimum at 0.09 at epoch 20 and then steadily climbs up and up to 0.25 at epoch 30.\n",
        "\n",
        "a. Explain why this is occuring.\n",
        "\n",
        "b. Describe what you should set your epoch hyperparameter to in this situation and why. \n",
        "\n",
        "c. Explain three separate techniques on how to combat this phenomenon.\n",
        "\n",
        "d. Implement one of these techniques in your current model (exluding data augmentation as this is already done)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xx1gpeIifUb_"
      },
      "source": [
        "*Solutions*\r\n",
        "\r\n",
        "\r\n",
        "1)\r\n",
        "Because the original model is trained on imagenet, which is a totally different distribution comparing to our dataset. It would not only take a long time for the network to learn $\\gamma$ and $\\beta$ in each batch normalization layer of the original architecture if it was not frozen, but also it was trained on a large dataset and would help our classification model, since our dataset of X-ray pictures is small and not enough to capture enough variances of the underlying distribution.\r\n",
        "\r\n",
        "\r\n",
        "2)\r\n",
        "\r\n",
        "a. The model started to overfit on the training data after 20 epochs.\r\n",
        "\r\n",
        "b. Set epoch to 20, so that the training would stop before overfitting. Epoch 20 gives the lowest validation loss.\r\n",
        "\r\n",
        "c. \r\n",
        "1. Add batch normalization layers to recenter the inputs to prevent too much diverging;\r\n",
        "2. Augment the training data so that it captures more variances of the underlying distribution, which may occur in the validation set that are not in our current training set;\r\n",
        "3. Add an early stopping callback which monitors the validation loss, so that it can stop the training automatically when it hits the lowest val_loss.\r\n",
        "\r\n",
        "d. Done.\r\n",
        "\r\n"
      ]
    }
  ]
}