t81_558_class_10_4_captioning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LXSZ9CRGevw0"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 10: Time Series in Keras**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3LFASLt5evw2"
   },
   "source": [
    "# Module 10 Material\n",
    "\n",
    "* Part 10.1: Time Series Data Encoding for Deep Learning [[Video]](https://www.youtube.com/watch?v=dMUmHsktl04&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_1_timeseries.ipynb)\n",
    "* Part 10.2: Programming LSTM with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=wY0dyFgNCgY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_2_lstm.ipynb)\n",
    "* Part 10.3: Text Generation with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=6ORnRAz3gnA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_3_text_generation.ipynb)\n",
    "* **Part 10.4: Image Captioning with Keras and TensorFlow** [[Video]](https://www.youtube.com/watch?v=NmoW_AYWkb4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_4_captioning.ipynb)\n",
    "* Part 10.5: Temporal CNN in Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=i390g8acZwk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_5_temporal_cnn.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7xH2XSZmevxx"
   },
   "source": [
    "# Part 10.4: Image Captioning with Keras and TensorFlow\n",
    "\n",
    "Image captioning is a new technology that combines LSTM text generation with the computer vision powers of a convolutional neural network.  I first saw this technology in [Andrej Karpathy's Dissertation](https://cs.stanford.edu/people/karpathy/main.pdf). Images from his work are shown here:\n",
    "\n",
    "![Captioning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/karpathy_thesis.jpg \"Captioning\")\n",
    "\n",
    "In this part, we will make use of LSTM and CNN to create a basic image captioning system. Transfer learning will be used to bring in these two projects:\n",
    "\n",
    "* [InceptionV3](https://arxiv.org/abs/1512.00567)\n",
    "* [Glove](https://nlp.stanford.edu/projects/glove/)\n",
    "\n",
    "Inception is used to extract features from the images.  Glove is a set of Natural Language Processing (NLP) vectors for common words.\n",
    "\n",
    "The following diagram gives a high-level overview of captioning.\n",
    "\n",
    "![Captioning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/caption-1.png \"Captioning\")\n",
    "\n",
    "We begin by importing the needed libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "m2p8iXQygqRN",
    "outputId": "8221ab8f-f57d-48fb-b14f-6539b5c2b0dc"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import string\n",
    "import glob\n",
    "from tensorflow.keras.applications import MobileNet\n",
    "import tensorflow.keras.applications.mobilenet  \n",
    "\n",
    "from tensorflow.keras.applications.inception_v3 import InceptionV3\n",
    "import tensorflow.keras.applications.inception_v3\n",
    "\n",
    "\n",
    "from tqdm import tqdm\n",
    "import tensorflow.keras.preprocessing.image\n",
    "import pickle\n",
    "from time import time\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import LSTM, Embedding, TimeDistributed, Dense, RepeatVector,\\\n",
    "                         Activation, Flatten, Reshape, concatenate, Dropout, BatchNormalization\n",
    "from tensorflow.keras.optimizers import Adam, RMSprop\n",
    "from tensorflow.keras import Input, layers\n",
    "from tensorflow.keras import optimizers\n",
    "\n",
    "from tensorflow.keras.models import Model\n",
    "\n",
    "from tensorflow.keras.layers import add\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "START = \"startseq\"\n",
    "STOP = \"endseq\"\n",
    "EPOCHS = 10\n",
    "USE_INCEPTION = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kEMwH956gaOG"
   },
   "source": [
    "The following function is used to nicely format elapsed times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mGt4swcqIFKR"
   },
   "outputs": [],
   "source": [
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TgTP26RygaOe"
   },
   "source": [
    "### Needed Data\n",
    "\n",
    "You will need to download the following data and place it in a folder for this example.  Point the *root_captioning* string at the folder that you are using for the caption generation. This folder should have the following sub-folders.\n",
    "\n",
    "* data - Create this directory to hold saved models.\n",
    "* [glove.6B](https://nlp.stanford.edu/projects/glove/) - Glove embeddings.\n",
    "* [Flicker8k_Dataset](https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_Dataset.zip) - Flicker dataset.\n",
    "* [Flicker8k_Text](https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_text.zip)\n",
    "\n",
    "Note, the original Flickr datasets are no longer available, but can be downloaded from [this article](https://machinelearningmastery.com/prepare-photo-caption-dataset-training-deep-learning-model/). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_7jgA6i88Iv8"
   },
   "source": [
    "### Google CoLab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m_vKum_ygaOS"
   },
   "source": [
    "If you are using Google CoLab then you will need to execute this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "id": "2W5oWBSofR7d",
    "outputId": "d0c0cac0-2068-414c-cb33-7c842e9dba0e"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "root_captioning = \"/content/drive/My Drive/projects/captions\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hj4K-WHafser"
   },
   "outputs": [],
   "source": [
    "root_captioning = \"/content/drive/My Drive/projects/captions\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VlBS8URRgaOo"
   },
   "source": [
    "### Clean/Build Dataset From Flickr8k\n",
    "\n",
    "We must pull in the Flickr dataset captions and clean them of extra whitespace, punctuation, and other distractions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ra-d7BLlf1nC"
   },
   "outputs": [],
   "source": [
    "null_punct = str.maketrans('', '', string.punctuation)\n",
    "lookup = dict()\n",
    "\n",
    "with open( os.path.join(root_captioning,'Flickr8k_text','Flickr8k.token.txt'), 'r') as fp:\n",
    "  \n",
    "  max_length = 0\n",
    "  for line in fp.read().split('\\n'):\n",
    "    tok = line.split()\n",
    "    if len(line) >= 2:\n",
    "      id = tok[0].split('.')[0]\n",
    "      desc = tok[1:]\n",
    "      \n",
    "      # Cleanup description\n",
    "      desc = [word.lower() for word in desc]\n",
    "      desc = [w.translate(null_punct) for w in desc]\n",
    "      desc = [word for word in desc if len(word)>1]\n",
    "      desc = [word for word in desc if word.isalpha()]\n",
    "      max_length = max(max_length,len(desc))\n",
    "      \n",
    "      if id not in lookup:\n",
    "        lookup[id] = list()\n",
    "      lookup[id].append(' '.join(desc))\n",
    "      \n",
    "lex = set()\n",
    "for key in lookup:\n",
    "  [lex.update(d.split()) for d in lookup[key]]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xdFUtyIPgaOy"
   },
   "source": [
    "Stats on what was collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "Gnq3ZjaSU79-",
    "outputId": "43b4d872-97a7-42ed-e573-119a323b973d"
   },
   "outputs": [],
   "source": [
    "print(len(lookup)) # How many unique words\n",
    "print(len(lex)) # The dictionary\n",
    "print(max_length) # Maximum length of a caption (in words)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jk1ts5ZfgaO5"
   },
   "source": [
    "Load the Glove embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JYVZ83DLsb1r"
   },
   "outputs": [],
   "source": [
    "# Warning, running this too soon on GDrive can sometimes not work.\n",
    "# Just rerun if len(img) = 0\n",
    "img = glob.glob(os.path.join(root_captioning,'Flicker8k_Dataset', '*.jpg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vzVqUkNGgaPG"
   },
   "source": [
    "Display the count of how many Glove embeddings we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "DosvCz6bVZA7",
    "outputId": "13c56fa5-4ef5-4d2e-d73e-0c36fdf30b3a"
   },
   "outputs": [],
   "source": [
    "len(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EXpBd3PrgaPL"
   },
   "source": [
    "Read all image names and use the predefined train/test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lhJSo7czkW03"
   },
   "outputs": [],
   "source": [
    "train_images_path = os.path.join(root_captioning,'Flickr8k_text','Flickr_8k.trainImages.txt') \n",
    "train_images = set(open(train_images_path, 'r').read().strip().split('\\n'))\n",
    "test_images_path = os.path.join(root_captioning,'Flickr8k_text','Flickr_8k.testImages.txt') \n",
    "test_images = set(open(test_images_path, 'r').read().strip().split('\\n'))\n",
    "\n",
    "train_img = []\n",
    "test_img = []\n",
    "\n",
    "for i in img:\n",
    "  f = os.path.split(i)[-1]\n",
    "  if f in train_images: \n",
    "    train_img.append(f) \n",
    "  elif f in test_images:\n",
    "    test_img.append(f) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aP2-k6XfgaPT"
   },
   "source": [
    "Display the size of the train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "ZZQZkT8X_qNt",
    "outputId": "56bbcab9-97ab-4fc0-9c2d-b1f16e3b8abe"
   },
   "outputs": [],
   "source": [
    "print(len(train_images))\n",
    "print(len(test_images))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iawCdtU4gaPa"
   },
   "source": [
    "Build the sequences.  We include a **start** and **stop** token at the beginning/end.  We will later use the **start** token to begin the process of generating a caption.  Encountering the **stop** token in the generated text will let us know we are done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cBX6TJbPF0nD"
   },
   "outputs": [],
   "source": [
    "train_descriptions = {k:v for k,v in lookup.items() if f'{k}.jpg' in train_images}\n",
    "for n,v in train_descriptions.items(): \n",
    "  for d in range(len(v)):\n",
    "    v[d] = f'{START} {v[d]} {STOP}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XHD6IZ3HgaPk"
   },
   "source": [
    "See how many discriptions were extracted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "v3_PCjddHcUe",
    "outputId": "ac0bb3b2-7ac6-4e9d-fa45-e89782220750"
   },
   "outputs": [],
   "source": [
    "len(train_descriptions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SAYpvjpogaPs"
   },
   "source": [
    "### Choosing a Computer Vision Neural Network to Transfer\n",
    "\n",
    "There are two neural networks that are accessed via transfer learning.  In this example, I use Glove for the text embedding and InceptionV3 to extract features from the images.  Both of these transfers serve to extract features from the raw text and the images.  Without this prior knowldge transferred in, this example would take consideraby more training.\n",
    "\n",
    "I made it so you can interchange the neural network used for the images.  By setting the values WIDTH, HEIGHT, and OUTPUT_DIM you can interchange images.  One characteristic that you are seeking for the image neural network is that it does not have too many outputs (once you strip the 1000-class imagenet classifier, as is common in transfer learning).  InceptionV3 has 2,048 features below the classifier and MobileNet has over 50K.  If the additional dimensions truely capture aspects of the images, then they are worthwhile.  However, having 50K features increases the processing needed and the complexity of the neural network we are constructing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 360
    },
    "colab_type": "code",
    "id": "DWOP58lrtGwt",
    "outputId": "c6d637f7-37f4-433a-9563-f0772feda712"
   },
   "outputs": [],
   "source": [
    "if USE_INCEPTION:\n",
    "  encode_model = InceptionV3(weights='imagenet')\n",
    "  encode_model = Model(encode_model.input, encode_model.layers[-2].output)\n",
    "  WIDTH = 299\n",
    "  HEIGHT = 299\n",
    "  OUTPUT_DIM = 2048\n",
    "  preprocess_input = keras.applications.inception_v3.preprocess_input\n",
    "else:\n",
    "  encode_model = MobileNet(weights='imagenet',include_top=False)\n",
    "  WIDTH = 224\n",
    "  HEIGHT = 224\n",
    "  OUTPUT_DIM = 50176\n",
    "  preprocess_input = keras.applications.mobilenet.preprocess_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wz8_Fu2lgaP3"
   },
   "source": [
    "The summary for the chosen image neural network to be transfered is displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "_QQ-D9Rq9SwF",
    "outputId": "61bc934b-ad17-4ce9-8ccc-5fb0611bcc53"
   },
   "outputs": [],
   "source": [
    "encode_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FZCQH7nvhNdc"
   },
   "source": [
    "### Creating the Training Set\n",
    "\n",
    "We we need to encode the images to create the training set.  Later we will encode new images to present them for captioning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SeT4zJ8l9Ps9"
   },
   "outputs": [],
   "source": [
    "def encodeImage(img):\n",
    "  # Resize all images to a standard size (specified bythe image encoding network)\n",
    "  img = img.resize((WIDTH, HEIGHT), Image.ANTIALIAS)\n",
    "  # Convert a PIL image to a numpy array\n",
    "  x = keras.preprocessing.image.img_to_array(img)\n",
    "  # Expand to 2D array\n",
    "  x = np.expand_dims(x, axis=0)\n",
    "  # Perform any preprocessing needed by InceptionV3 or others\n",
    "  x = preprocess_input(x)\n",
    "  # Call InceptionV3 (or other) to extract the smaller feature set for the image.\n",
    "  x = encode_model.predict(x) # Get the encoding vector for the image\n",
    "  # Shape to correct form to be accepted by LSTM captioning network.\n",
    "  x = np.reshape(x, OUTPUT_DIM )\n",
    "  return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ni6VXDiVgaQe"
   },
   "source": [
    "We can how generate the training set.  This will involve looping over every JPG that was provied.  Because this can take awhile to perform we will save it to a pickle file.  This saves the considerable time needed to completly reprocess all of the images.  Because the images are processed differently by different transferred neural networks, the output dimensions are also made part of the file name.  If you changed from InceptionV3 to MobileNet, the number of output dimensions would change, and a new file would be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OfusVb6uKREw"
   },
   "outputs": [],
   "source": [
    "train_path = os.path.join(root_captioning,\"data\",f'train{OUTPUT_DIM}.pkl')\n",
    "if not os.path.exists(train_path):\n",
    "  start = time()\n",
    "  encoding_train = {}\n",
    "  for id in tqdm(train_img):\n",
    "    image_path = os.path.join(root_captioning,'Flicker8k_Dataset', id)\n",
    "    img = keras.preprocessing.image.load_img(image_path, target_size=(HEIGHT, WIDTH))\n",
    "    encoding_train[id] = encodeImage(img)\n",
    "  with open(train_path, \"wb\") as fp:\n",
    "    pickle.dump(encoding_train, fp)\n",
    "  print(f\"\\nGenerating training set took: {hms_string(time()-start)}\")\n",
    "else:\n",
    "  with open(train_path, \"rb\") as fp:\n",
    "    encoding_train = pickle.load(fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0sM7abSfgaQo"
   },
   "source": [
    "A similar process must also be performed for the test images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mO2cNDOP64-"
   },
   "outputs": [],
   "source": [
    "test_path = os.path.join(root_captioning,\"data\",f'test{OUTPUT_DIM}.pkl')\n",
    "if not os.path.exists(test_path):\n",
    "  start = time()\n",
    "  encoding_test = {}\n",
    "  for id in tqdm(test_img):\n",
    "    image_path = os.path.join(root_captioning,'Flicker8k_Dataset', id)\n",
    "    img = keras.preprocessing.image.load_img(image_path, target_size=(HEIGHT, WIDTH))\n",
    "    encoding_test[id] = encodeImage(img)\n",
    "  with open(test_path, \"wb\") as fp:\n",
    "    pickle.dump(encoding_test, fp)\n",
    "  print(f\"\\nGenerating testing set took: {hms_string(time()-start)}\")\n",
    "else:\n",
    "  with open(test_path, \"rb\") as fp:\n",
    "    encoding_test = pickle.load(fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Ao7N4JngaQv"
   },
   "source": [
    "Next we separate the captions that will be usef for training.  There are two sides to this training, the images and the captions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "C8DFzzPmXg-n",
    "outputId": "ca327a2d-2588-48e8-9954-f588b391b0f9"
   },
   "outputs": [],
   "source": [
    "all_train_captions = []\n",
    "for key, val in train_descriptions.items():\n",
    "    for cap in val:\n",
    "        all_train_captions.append(cap)\n",
    "len(all_train_captions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KiTjOC_9x1_x"
   },
   "source": [
    "Words that do not occur very often can be misleading to neural network training.  It is better to simply remove such words.  Here we remove any words that occur less than 10 times.  We display what the total vocabulary shrunk to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "-HxHSgLgYUUE",
    "outputId": "39c00534-7778-41b4-8829-9ad867af9a30"
   },
   "outputs": [],
   "source": [
    "word_count_threshold = 10\n",
    "word_counts = {}\n",
    "nsents = 0\n",
    "for sent in all_train_captions:\n",
    "    nsents += 1\n",
    "    for w in sent.split(' '):\n",
    "        word_counts[w] = word_counts.get(w, 0) + 1\n",
    "\n",
    "vocab = [w for w in word_counts if word_counts[w] >= word_count_threshold]\n",
    "print('preprocessed words %d ==> %d' % (len(word_counts), len(vocab)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rj9g0NtXz7Kd"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E3vQYH860N8J"
   },
   "source": [
    "Next we build two lookup tables for this vocabulary. One idxtoword convers index numbers to actual words to index values.  The wordtoidx lookup table performs the opposit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Oi3zqAUhYcNP",
    "outputId": "d60c25a4-0a22-4309-a65d-634fef05ade1"
   },
   "outputs": [],
   "source": [
    "idxtoword = {}\n",
    "wordtoidx = {}\n",
    "\n",
    "ix = 1\n",
    "for w in vocab:\n",
    "    wordtoidx[w] = ix\n",
    "    idxtoword[ix] = w\n",
    "    ix += 1\n",
    "    \n",
    "vocab_size = len(idxtoword) + 1 \n",
    "vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rxs9sfOq0Xxz"
   },
   "source": [
    "Previously we added a start and stop token to all sentences.  We must account for this in the maximum length of captions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "KSohY53bYjTi",
    "outputId": "745ba047-bc40-44fa-c8df-f0b668e9e5cb"
   },
   "outputs": [],
   "source": [
    "max_length +=2\n",
    "print(max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YOLTDiQC0i_r"
   },
   "source": [
    "### Using a Data Generator\n",
    "\n",
    "Up to this point we've always generated training data ahead of time and fit the neural network to it.  It is not always practical to generate all of the training data ahead of time.  The memory demands can be considerable.  If the training data can be generated, as the neural network needs it, it is possable to use a Keras generator.  The generator will create new data, as it is needed.  The generator provided here creates the training data for the caption neural network, as it is needed.\n",
    "\n",
    "If we were to build all needed training data ahead of time it would look something like below.\n",
    "\n",
    "![Captioning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/caption-2.png \"Captioning\")\n",
    "\n",
    "Here we are just training on two captions.  However, we would have to duplicate the image for each of these partial captions that we have.  Additionally the Flikr8K data set has 5 captions for each picture.  Those would all require duplication of data as well.  It is much more efficient to just generate the data as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gZsbbCx6d04X"
   },
   "outputs": [],
   "source": [
    "def data_generator(descriptions, photos, wordtoidx, max_length, num_photos_per_batch):\n",
    "  # x1 - Training data for photos\n",
    "  # x2 - The caption that goes with each photo\n",
    "  # y - The predicted rest of the caption\n",
    "  x1, x2, y = [], [], []\n",
    "  n=0\n",
    "  while True:\n",
    "    for key, desc_list in descriptions.items():\n",
    "      n+=1\n",
    "      photo = photos[key+'.jpg']\n",
    "      # Each photo has 5 descriptions\n",
    "      for desc in desc_list:\n",
    "        # Convert each word into a list of sequences.\n",
    "        seq = [wordtoidx[word] for word in desc.split(' ') if word in wordtoidx]\n",
    "        # Generate a training case for every possible sequence and outcome\n",
    "        for i in range(1, len(seq)):\n",
    "          in_seq, out_seq = seq[:i], seq[i]\n",
    "          in_seq = pad_sequences([in_seq], maxlen=max_length)[0]\n",
    "          out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]\n",
    "          x1.append(photo)\n",
    "          x2.append(in_seq)\n",
    "          y.append(out_seq)\n",
    "      if n==num_photos_per_batch:\n",
    "        yield [[np.array(x1), np.array(x2)], np.array(y)]\n",
    "        x1, x2, y = [], [], []\n",
    "        n=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vtIIFt4m6pTv"
   },
   "source": [
    "### Loading Glove Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "IY_9XZ4Hec73",
    "outputId": "f432653b-3506-4602-df3e-8c9484622cba"
   },
   "outputs": [],
   "source": [
    "glove_dir = os.path.join(root_captioning,'glove.6B')\n",
    "embeddings_index = {} \n",
    "f = open(os.path.join(glove_dir, 'glove.6B.200d.txt'), encoding=\"utf-8\")\n",
    "\n",
    "for line in tqdm(f):\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "\n",
    "f.close()\n",
    "print(f'Found {len(embeddings_index)} word vectors.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kYH3-S2n82MN"
   },
   "source": [
    "### Building the Neural Network\n",
    "\n",
    "An embedding matrix is built from Glove.  This will be directly copied to the weight matrix of the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "adEmnYFEfog6"
   },
   "outputs": [],
   "source": [
    "embedding_dim = 200\n",
    "\n",
    "# Get 200-dim dense vector for each of the 10000 words in out vocabulary\n",
    "embedding_matrix = np.zeros((vocab_size, embedding_dim))\n",
    "\n",
    "for word, i in wordtoidx.items():\n",
    "    #if i < max_words:\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # Words not found in the embedding index will be all zeros\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GpuNLhoaCf5N"
   },
   "source": [
    "The matrix makes sense.  It is 1652 (the size of the vocabulary) by 200 (the number of features Glove generates for each word)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "86hPHHjSfx8p",
    "outputId": "890f7079-2206-4be0-85d4-0372f4b28ebb"
   },
   "outputs": [],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139
    },
    "colab_type": "code",
    "id": "2homy6Znf7wL",
    "outputId": "42e5d601-fc30-401b-a3f7-85db628f4a3f"
   },
   "outputs": [],
   "source": [
    "inputs1 = Input(shape=(OUTPUT_DIM,))\n",
    "fe1 = Dropout(0.5)(inputs1)\n",
    "fe2 = Dense(256, activation='relu')(fe1)\n",
    "inputs2 = Input(shape=(max_length,))\n",
    "se1 = Embedding(vocab_size, embedding_dim, mask_zero=True)(inputs2)\n",
    "se2 = Dropout(0.5)(se1)\n",
    "se3 = LSTM(256)(se2)\n",
    "decoder1 = add([fe2, se3])\n",
    "decoder2 = Dense(256, activation='relu')(decoder1)\n",
    "outputs = Dense(vocab_size, activation='softmax')(decoder2)\n",
    "caption_model = Model(inputs=[inputs1, inputs2], outputs=outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "UcCe2n5pnGfo",
    "outputId": "c95db51a-8e4d-47ff-f8aa-213378be902e"
   },
   "outputs": [],
   "source": [
    "embedding_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 493
    },
    "colab_type": "code",
    "id": "asGo7TqRfgJp",
    "outputId": "b8f60753-d6a0-4eca-b367-8e0ed32c30ec"
   },
   "outputs": [],
   "source": [
    "caption_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "gze36nMIfoEg",
    "outputId": "81ded466-db0f-4c7e-f400-29253af8d719"
   },
   "outputs": [],
   "source": [
    "caption_model.layers[2].set_weights([embedding_matrix])\n",
    "caption_model.layers[2].trainable = False\n",
    "caption_model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wCSzCGtB8-KM"
   },
   "source": [
    "### Train the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0LXG3PE5fxzU"
   },
   "outputs": [],
   "source": [
    "number_pics_per_bath = 3\n",
    "steps = len(train_descriptions)//number_pics_per_bath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dyCHBbjnf2b1"
   },
   "outputs": [],
   "source": [
    "model_path = os.path.join(root_captioning,\"data\",f'caption-model.hdf5')\n",
    "if not os.path.exists(model_path):\n",
    "  for i in tqdm(range(EPOCHS*2)):\n",
    "      generator = data_generator(train_descriptions, encoding_train, wordtoidx, max_length, number_pics_per_bath)\n",
    "      caption_model.fit_generator(generator, epochs=1, steps_per_epoch=steps, verbose=1)\n",
    "\n",
    "  caption_model.optimizer.lr = 1e-4\n",
    "  number_pics_per_bath = 6\n",
    "  steps = len(train_descriptions)//number_pics_per_bath\n",
    "\n",
    "  for i in range(EPOCHS):\n",
    "      generator = data_generator(train_descriptions, encoding_train, wordtoidx, max_length, number_pics_per_bath)\n",
    "      caption_model.fit_generator(generator, epochs=1, steps_per_epoch=steps, verbose=1)  \n",
    "  caption_model.save_weights(model_path)\n",
    "  print(f\"\\Training took: {hms_string(time()-start)}\")\n",
    "else:\n",
    "  caption_model.load_weights(model_path)\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ymx07aUHpHMZ"
   },
   "source": [
    "### Generating Captions\n",
    "\n",
    "It is important to understand that a caption is not generated with one single call to the neural network's predict function.  Neural networks output a fixed-length tensor.  To get a variable length output, such as free-form text, requires multiple calls to the neural network.\n",
    "\n",
    "The neural network accepts two objects (which are mapped to the input neurons).  The first is the photo.  The second is an ever growing caption.  The caption begins with just the starting token.  The neural network's output is the prediction of the next word in the caption.  This continues until an end token is predicted or we reach the maximum length of a caption.  Each time predict a new word is predicted for the caption.  The word that has the highest probability (from the neural network) is chosen. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JPKINJ3JMbSy"
   },
   "outputs": [],
   "source": [
    "def generateCaption(photo):\n",
    "    in_text = START\n",
    "    for i in range(max_length):\n",
    "        sequence = [wordtoidx[w] for w in in_text.split() if w in wordtoidx]\n",
    "        sequence = pad_sequences([sequence], maxlen=max_length)\n",
    "        yhat = caption_model.predict([photo,sequence], verbose=0)\n",
    "        yhat = np.argmax(yhat)\n",
    "        word = idxtoword[yhat]\n",
    "        in_text += ' ' + word\n",
    "        if word == STOP:\n",
    "            break\n",
    "    final = in_text.split()\n",
    "    final = final[1:-1]\n",
    "    final = ' '.join(final)\n",
    "    return final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MAv_Bab17WeD"
   },
   "source": [
    "### Evaluate Performance on Test Data from Flicker8k\n",
    "\n",
    "The caption model performs relativly well on images that are similar to what it trained on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "TtaaQI8wMsst",
    "outputId": "77525180-c6a8-4e2d-c0dd-77b089948343"
   },
   "outputs": [],
   "source": [
    "for z in range(10):\n",
    "  pic = list(encoding_test.keys())[z]\n",
    "  image = encoding_test[pic].reshape((1,OUTPUT_DIM))\n",
    "  print(os.path.join(root_captioning,'Flicker8k_Dataset', pic))\n",
    "  x=plt.imread(os.path.join(root_captioning,'Flicker8k_Dataset', pic))\n",
    "  plt.imshow(x)\n",
    "  plt.show()\n",
    "  print(\"Caption:\",generateCaption(image))\n",
    "  print(\"_____________________________________\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "a1mncR2JPno-",
    "outputId": "baf1f945-da4c-41e9-85c1-8e270d214350"
   },
   "outputs": [],
   "source": [
    "encoding_test[pic].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HmPT4uNu7hgN"
   },
   "source": [
    "### Evaluate Performance on My Own Photos \n",
    "\n",
    "In the \"photos\" folder of this GitHub repository I keep a collection of personal photos that I like to test neural networks on.  These are completly separate from the Flicker8K dataset and as a result, the caption neural network does not perform nearly as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "-jl0BUNIMxGv",
    "outputId": "f22f79d2-4645-476b-f953-11a53be5ad0e"
   },
   "outputs": [],
   "source": [
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "from io import BytesIO\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "urls = [\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/2015-03-09-phd-2nd-cluster-visit-1.png?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/about-jeff-heaton-2018.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/annie_dog.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/bread_n_breakfast.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/family_christmas.png?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/github_rock.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory.jpeg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_coat.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_exercise.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_home.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_n_stuffed.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_run.jpeg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/hickory_sleep.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/jeff_books.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/jeff_cook.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/jeff_cube.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/jeff_laptop.jpg?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/jeff_laptops.png?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/road.JPG?raw=true\",\n",
    "  \"https://github.com/jeffheaton/t81_558_deep_learning/blob/master/photos/snow_shovel.jpg?raw=true\"\n",
    "]\n",
    "\n",
    "for url in urls:\n",
    "  response = requests.get(url)\n",
    "  img = Image.open(BytesIO(response.content))\n",
    "  img.load()\n",
    "\n",
    "  plt.imshow(img)\n",
    "  plt.show()\n",
    "  \n",
    "  response = requests.get(url)\n",
    "\n",
    "  img = encodeImage(img).reshape((1,OUTPUT_DIM))\n",
    "  print(img.shape)\n",
    "  print(\"Caption:\",generateCaption(img))\n",
    "  print(\"_____________________________________\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFZpd0AZevxz"
   },
   "source": [
    "# Module 10 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 10](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class10.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "orKUqz6zGeiB"
   },
   "outputs": [],
   "source": [
    "caption_model.save_weights(\"test.hdf5\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of Copy of t81_558_class10_3_captioning.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}