4.4.3_NLP_unsupervised.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction  \n",
    "This notebook analyzes paragraphs from the copus 'Emma' in order to identify sentences that are similar among a group of 10 sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean and process the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package gutenberg to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package gutenberg is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[ Emma by Jane Austen 1816 ]', 'VOLUME I', 'CHAPTER I', 'Emma Woodhouse , handsome , clever , and rich , with a comfortable home and happy disposition , seemed to unite some of the best blessings of existence ; and had lived nearly twenty - one years in the world with very little to distress or vex her .']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import gutenberg\n",
    "nltk.download('punkt')\n",
    "nltk.download('gutenberg')\n",
    "import re\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#reading in the data, this time in the form of paragraphs\n",
    "emma = gutenberg.paras('austen-emma.txt')\n",
    "\n",
    "#processing\n",
    "emma_paras = []\n",
    "for paragraph in emma:\n",
    "#    print(paragraph)\n",
    "\n",
    "    para = paragraph[0]\n",
    "    \n",
    "    #removing the double-dash from all words\n",
    "    para = [re.sub(r'--', '', word) for word in para]\n",
    "    \n",
    "    #Forming each paragraph into a string and adding it to the list of strings.\n",
    "    emma_paras.append(' '.join(para))\n",
    " #   print(para)\n",
    "\n",
    "print(emma_paras[0:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the Model  \n",
    "\n",
    "#### Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 1020\n",
      "Original sentence: A very few minutes more , however , completed the present trial .\n",
      "Tf_idf vector: {'minutes': 0.7127450310382584, 'present': 0.701423210857947}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "X_train, X_test = train_test_split(emma_paras, test_size=0.4, random_state=0)\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=0.5, # drop words that occur in more than half the paragraphs\n",
    "                             min_df=4, # only use words that appear at least twice\n",
    "                             stop_words='english', \n",
    "                             lowercase=True, #convert everything to lower case (since Alice in Wonderland has the HABIT of CAPITALIZING WORDS for EMPHASIS)\n",
    "                             use_idf=True,# we definitely want to use inverse document frequencies in our weighting\n",
    "                             norm=u'l2', # Applies a correction factor so that longer paragraphs and shorter paragraphs get treated equally\n",
    "                             smooth_idf=True # Adds 1 to all document frequencies, as if an extra document existed that used every word once.  Prevents divide-by-zero errors\n",
    "                            )\n",
    "\n",
    "\n",
    "#Applying the vectorizer\n",
    "emma_paras_tfidf=vectorizer.fit_transform(emma_paras)\n",
    "print(\"Number of features: %d\" % emma_paras_tfidf.get_shape()[1])\n",
    "\n",
    "#splitting into training and test sets\n",
    "X_train_tfidf, X_test_tfidf= train_test_split(emma_paras_tfidf, test_size=0.4, random_state=0)\n",
    "\n",
    "#Reshapes the vectorizer output into something people can read\n",
    "X_train_tfidf_csr = X_train_tfidf.tocsr()\n",
    "\n",
    "#number of paragraphs\n",
    "n = X_train_tfidf_csr.shape[0]\n",
    "\n",
    "#A list of dictionaries, one per paragraph\n",
    "tfidf_bypara = [{} for _ in range(0,n)]\n",
    "\n",
    "#List of features\n",
    "terms = vectorizer.get_feature_names()\n",
    "\n",
    "#for each paragraph, lists the feature words and their tf-idf scores\n",
    "for i, j in zip(*X_train_tfidf_csr.nonzero()):\n",
    "    tfidf_bypara[i][terms[j]] = X_train_tfidf_csr[i, j]\n",
    "\n",
    "#Keep in mind that the log base 2 of 1 is 0, so a tf-idf score of 0 indicates that the word was present once in that sentence.\n",
    "print('Original sentence:', X_train[5])\n",
    "print('Tf_idf vector:', tfidf_bypara[5])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the model on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "949\n",
      "Original sentence: \" And I am quite serious too , I assure you ,\" replied Mrs . Elton gaily , \" in resolving to be always on the watch , and employing my friends to watch also , that nothing really unexceptionable may pass us .\"\n",
      "Tf_idf vector: {'unexceptionable': 0.34021006165384615, 'assure': 0.2849583443549245, 'friends': 0.26771969260442585, 'quite': 0.20397979981522793, 'replied': 0.21622713776874078, 'watch': 0.6630989315438113, 'elton': 0.17960684463226512, 'pass': 0.302756944818224, 'really': 0.23377239488085674, 'mrs': 0.1573212521757836}\n"
     ]
    }
   ],
   "source": [
    "#Reshapes the vectorizer output into something people can read\n",
    "X_test_tfidf_csr = X_test_tfidf.tocsr()\n",
    "\n",
    "#number of paragraphs\n",
    "n_test = X_test_tfidf_csr.shape[0]\n",
    "print(n_test)\n",
    "\n",
    "#A list of dictionaries, one per paragraph\n",
    "tfidf_bypara_test = [{} for _ in range(0,n_test)]\n",
    "\n",
    "#List of features\n",
    "terms_test = vectorizer.get_feature_names()\n",
    "\n",
    "#for each paragraph, lists the feature words and their tf-idf scores\n",
    "for i, j in zip(*X_test_tfidf_csr.nonzero()):\n",
    "    tfidf_bypara_test[i][terms[j]] = X_test_tfidf_csr[i, j]\n",
    "\n",
    "#Keep in mind that the log base 2 of 1 is 0, so a tf-idf score of 0 indicates that the word was present once in that sentence.\n",
    "print('Original sentence:', X_test[5])\n",
    "print('Tf_idf vector:', tfidf_bypara_test[5])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reduce the number of features on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percent variance captured by all components: 49.98315683637081\n",
      "Component 0:\n",
      "\" Oh !         0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Oh !\"        0.999918\n",
      "\" Oh !         0.999918\n",
      "\" Me ! oh !    0.999918\n",
      "\" Oh !         0.999918\n",
      "Name: 0, dtype: float64\n",
      "Component 1:\n",
      "\" Well , Mrs . Weston ,\" said Emma triumphantly when he left them , \" what do you say now to Mr . Knightley ' s marrying Jane Fairfax ?\"                                                                                                                                                                                                                                                                                                             0.626247\n",
      "After tea , Mr . and Mrs . Weston , and Mr . Elton sat down with Mr . Woodhouse to cards .                                                                                                                                                                                                                                                                                                                                                           0.600988\n",
      "Frank turned instantly to Emma , to claim her former promise ; and boasted himself an engaged man , which his father looked his most perfect approbation of  and it then appeared that Mrs . Weston was wanting _him_ to dance with Mrs . Elton himself , and that their business was to help to persuade him into it , which was done pretty soon . Mr . Weston and Mrs . Elton led the way , Mr . Frank Churchill and Miss Woodhouse followed .    0.564293\n",
      "In this walk Emma and Mr . Weston found all the others assembled ; and towards this view she immediately perceived Mr . Knightley and Harriet distinct from the rest , quietly leading the way .                                                                                                                                                                                                                                                     0.553226\n",
      "Harriet , Mr . Elton , and Mr . Knightley , their own especial set , were the only persons invited to meet them ; the hours were to be early , as well as the numbers few ; Mr . Woodhouse ' s habits and inclination being consulted in every thing .                                                                                                                                                                                               0.552066\n",
      "\" I think , indeed ,\" said John Knightley pleasantly , \" that Mr . Weston has some little claim .                                                                                                                                                                                                                                                                                                                                                    0.529898\n",
      "While she was gone , Mr . Knightley called , and sat some time with Mr . Woodhouse and Emma , till Mr . Woodhouse , who had previously made up his mind to walk out , was persuaded by his daughter not to defer it , and was induced by the entreaties of both , though against the scruples of his own civility , to leave Mr . Knightley for that purpose .                                                                                       0.527057\n",
      "\" Mr .                                                                                                                                                                                                                                                                                                                                                                                                                                               0.516451\n",
      "\" Mrs . Weston ' s manners ,\" said Emma , \" were always particularly good .                                                                                                                                                                                                                                                                                                                                                                          0.516140\n",
      "\" He is a person I never think of from one month ' s end to another ,\" said Mr . Knightley , with a degree of vexation , which made Emma immediately talk of something else , though she could not comprehend why he should be angry .                                                                                                                                                                                                               0.515216\n",
      "Name: 1, dtype: float64\n",
      "Component 2:\n",
      "CHAPTER XVIII    1.0\n",
      "CHAPTER XIX      1.0\n",
      "CHAPTER VIII     1.0\n",
      "CHAPTER XII      1.0\n",
      "CHAPTER XV       1.0\n",
      "CHAPTER XIV      1.0\n",
      "CHAPTER VI       1.0\n",
      "CHAPTER XVII     1.0\n",
      "CHAPTER XVIII    1.0\n",
      "CHAPTER XVII     1.0\n",
      "Name: 2, dtype: float64\n",
      "Component 3:\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !\"    0.996645\n",
      "\" Ah !     0.996645\n",
      "\" Ah !     0.996645\n",
      "Name: 3, dtype: float64\n",
      "Component 4:\n",
      "\" My Emma !\"                                                                                                                                        0.741789\n",
      "\" Emma !\"                                                                                                                                           0.741789\n",
      "Emma wondered on what , of all the medley , she would fix .                                                                                         0.741789\n",
      "Emma had done .                                                                                                                                     0.741789\n",
      "\" And I do envy him , Emma .                                                                                                                        0.741789\n",
      "Emma was extremely confused .                                                                                                                       0.645629\n",
      "\" Such an imagination has crossed me , I own , Emma ; and if it never occurred to you before , you may as well take it into consideration now .\"    0.644034\n",
      "Emma denied none of it aloud , and agreed to none of it in private .                                                                                0.610345\n",
      "Nobody had any information to give ; and , after a few more wonderings , Emma said ,                                                                0.565608\n",
      "It was as much as Emma could bear , without being impolite .                                                                                        0.561373\n",
      "Name: 4, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "#Our SVD data reducer.  We are going to reduce the feature space from 949 to 100.\n",
    "svd= TruncatedSVD(100)\n",
    "lsa = make_pipeline(svd, Normalizer(copy=False))\n",
    "\n",
    "# Run SVD on the training data, then project the training data.\n",
    "X_test_lsa = lsa.fit_transform(X_test_tfidf)\n",
    "\n",
    "variance_explained=svd.explained_variance_ratio_\n",
    "total_variance = variance_explained.sum()\n",
    "print(\"Percent variance captured by all components:\",total_variance*100)\n",
    "\n",
    "#Looking at what sorts of paragraphs our solution considers similar, for the first five identified topics\n",
    "paras_by_component=pd.DataFrame(X_test_lsa,index=X_test)\n",
    "for i in range(5):\n",
    "    print('Component {}:'.format(i))\n",
    "    print(paras_by_component.loc[:,i].sort_values(ascending=False)[0:10])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspect the sentence similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAF2VJREFUeJzt3X2UXVV5x/HvL5OEhIQklgQLSZCgAYzoIpBGlIpYUBN1kVq1gMsiLGBcLYhKX8TaBRXbLrUKxVV8iRAVRSgC1hSj4Bu+lUCCvEgC0TAgGQMGBIIRMJmZp3/cE3oZ595zb+aePeee/D6ss3Je97MvmTx3zz77nK2IwMzM0hg31hUwM9udOOmamSXkpGtmlpCTrplZQk66ZmYJOemamSXkpGtm1oCkFZK2SLq7wXFJ+qSkjZLuknR4XplOumZmjX0BWNLk+FJgfrb0Ap/OK9BJ18ysgYj4IfBYk1OWAZdHzWpghqR9m5U5vpMVHMmOR/uSPPJ28CFvSREmufHqSRarb+tDyWItmjk/SZxJ4wr/EX/WxqceThbr/VMOSxbroqfXJYvV9+jtGm0Z7eScibNe+C5qLdSdlkfE8jbCzQY21W33Z/sa/mNK9xNpZlYyWYJtJ8kON9KXRNOk76RrZtUyNJgyWj8wt257DrC52QXu0zWzahkcaH0ZvZXAydkohiOBrRHRtJ/OLV0zq5SIoY6VJelK4BhgpqR+4HxgQi1OfAZYBbwB2Ag8BZyaV6aTrplVy1Dnkm5EnJRzPIAz2ynTSdfMqqWDLd0iOOmaWbWkvZHWNiddM6uWbm/pSjqE2lMXs6mNP9sMrIyIewqum5lZ26IzoxIK03TImKT3A1dRGwB8K7AmW79S0rnFV8/MrE1DQ60vYyCvpXsa8JKI2FG/U9KFwDrgIyNdJKmX7NG6T33iXzj95KY3AM3MOqfLuxeGgP2AXw7bv292bET1j9aleveCmRnQ9TfS3gt8V9Iv+P+XOuwPvAg4q8iKmZntkm5u6UbEtyQdBCymdiNN1J41XhMR5f46MbPdU8lvpOWOXojaM3WrE9TFzGz0xugGWas8TtfMKqXsv4Q76ZpZtXRzn66ZWddx94KZWUJu6ZqZJTS4I/+cMeSka2bVsrt3L6SapXfDvdcmiZNaylmO95+2T7JYW7ZvTRYrlYnjJiSLlXKG3q77t+XuBTOzhHb3lq6ZWVJOumZm6YRvpJmZJeQ+XTOzhNy9YGaWkFu6ZmYJuaVrZpaQW7pmZgkNlPsl5k1nA25G0qmdrIiZWUfEUOvLGNjlpAt8qNEBSb2S1kpa++Qzj44ihJlZm7p5CnZJdzU6BDy/0XX1swEfOHOhZwM2s3S6vE/3+cDrgceH7Rfwv4XUyMxsNLp89ML1wNSIuGP4AUk3FVIjM7PRKHlLt2mfbkScFhE/bnDs7cVUycxsFAYGWl9ySFoiaYOkjZLOHeH4/pK+L+l2SXdJekNemaO5kWZmVj4RrS9NSOoBLgGWAguAkyQtGHbaPwFXR8RC4ETgU3nV8zhdM6uWzvXpLgY2RkQfgKSrgGXA+rpzApiWrU8HNucV6qRrZtXSRtKV1Av01u1ano2+ApgNbKo71g+8fFgR/wzcKOndwBTguLyYTrpmVi1t3EirH946Ao10ybDtk4AvRMQnJL0C+JKkQyMaV8JJ18yqZXCwUyX1A3Prtufwh90HpwFLACLiZkmTgJnAlkaFVuZGWsoJHM2sxDr3RNoaYL6keZImUrtRtnLYOQ8CxwJIejEwCXikWaGVael23YylZlaMDt1Ii4gBSWcBNwA9wIqIWCfpAmBtRKwE/hb4nKT3Uet6OCWi+bCIyiRdMzOgow9HRMQqYNWwfefVra8HjmqnTCddM6uUGCr3616cdM2sWrr83QtmZt2lc6MXCuGka2bV4paumVlCTrpmZgnlvMhmrDnpmlm1lLylm/tEmqRDJB0raeqw/UuKq5aZ2S4aitaXMdA06Uo6G/g68G7gbknL6g7/W5EVMzPbJYODrS9jIK974QzgiIjYJukA4BpJB0TExYz8Bh7gua9L23vKHKZNmtmh6pqZNRcl717IS7o9EbENICIekHQMtcT7ApokXc8GbGZjpuRPpOX16T4s6bCdG1kCfhO1V5e9tMiKmZntkhhqfRkDeS3dk4HnzN4WEQPAyZI+W1itzMx2Vclbuk2TbkT0Nzn2k85Xx8xslAb8GLCZWTpj1G3QKiddM6uWbu5eMDPrNt0+ZMzMrLu4pWtmltDunnTHq6foEEB1ZwNOOeFmyv+Hg5HmDvPknj2SxAE4fdLByWJ9ctudyWKl/Lnoe/T20Rfil5ibmaXjOdLMzFJy0jUzS8ijF8zMEnJL18wsISddM7N0YtDdC2Zm6bila2aWjoeMmZml1O1JV9JiICJijaQFwBLg3ohYVXjtzMzaVe4u3eZJV9L5wFJgvKRvAy8HbgLOlbQwIv61wXXPTky5z9T9mT5pVkcrbWbWSAyUO+vmzZH2VuAo4GjgTODPI+IC4PXACY0uiojlEbEoIhY54ZpZUkNtLDkkLZG0QdJGSec2OOcvJa2XtE7SV/LKzOteGIiIQeApSfdFxJMAEfG0pHJ/nZjZbqlTN9Ik9QCXAK8F+oE1klZGxPq6c+YDHwCOiojHJe2TV25eS3e7pD2z9SPqAk2n9D0nZrZb6lxLdzGwMSL6ImI7cBWwbNg5ZwCXRMTjABGxJa/QvKR7dEQ8lRVWX8UJwDtzq2xmllgMRcuLpF5Ja+uW3rqiZgOb6rb7s331DgIOkvQTSaslLcmrX95swL9vsP9R4NG8ws3Mkmvjd/CIWA4sb3BYI10ybHs8MB84BpgD/EjSoRHxRKOYHqdrZpUSAx0rqh+YW7c9B9g8wjmrI2IHcL+kDdSS8JpGheZ1L5iZdZUYan3JsQaYL2mepInAicDKYef8N/AaAEkzqXU39DUr1C1dM6uWDt3ij4gBSWcBNwA9wIqIWCfpAmBtRKzMjr1O0npgEPj7iPhNs3KddM2sUlpowbZeVu3J21XD9p1Xtx7AOdnSEiddM6uUTibdIhSedPu2PlR0CAD2n5Y7JrkrpZyJtYozD28beDpJHIArtt+fLNY+e8xIFuuxHduSxeqEGBxp0EF5uKVrZpWy27d0zcxSiiG3dM3MknFL18wsoQi3dM3MknFL18wsoSGPXjAzS8c30szMEip70m37hTeSLi+iImZmnRDR+jIW8iamHP5GHQGvkTQDICKOL6piZma7ouwt3bzuhTnAeuBSai/vFbAI+ESzi+pnA1bPdMaNmzL6mpqZtaDsQ8byuhcWAbcBHwS2RsRNwNMR8YOI+EGji+pnA3bCNbOUBgfV8jIW8qbrGQIukvTV7M9f511jZjaWyt7SbSmBRkQ/8DZJbwSeLLZKZma7rtv7dJ8jIr4BfKOgupiZjdpYjUpolbsKzKxSKtXSNTMru8Ghcs+366RrZpXi7gUzs4SGqjB6wcysW1RiyJiZWbfY7bsXFs2cX3QIALZs35okTmqDMZgsVhVnHv7RS85NEgdg6RM3J4v1qlkLksXqttmA3b1gZpaQRy+YmSVU8t4FJ10zqxZ3L5iZJeTRC2ZmCZV8MmAnXTOrlqDcLd1y3+YzM2vTQKjlJY+kJZI2SNooqeH4Q0lvlRSSFuWV6aRrZpUSqOWlGUk9wCXAUmABcJKkPxggLWkv4Gzgllbq11bSlfSnks6R9Lp2rjMzS2WojSXHYmBjRPRFxHbgKmDZCOd9GPgY8Ewr9WuadCXdWrd+BvCfwF7A+c2a2mZmY6Wdlq6kXklr65beuqJmA5vqtvuzfc+StBCYGxHXt1q/vBtpE+rWe4HXRsQjkj4OrAY+MtJF9bMBz5s+n3323K/V+piZjUo7oxciYjmwvMHhkfofnn32QtI44CLglDZC5nYvjJP0PEl7A4qIR7KK/g4YaHRR/WzATrhmltIgannJ0Q/MrdueA2yu294LOBS4SdIDwJHAyrybaXkt3enUpmAXEJL+OCIeljSVkb8FzMzGVAdn61kDzJc0D/gVcCLw9p0HI2IrMHPntqSbgL+LiLXNCs2bgv2ABoeGgDe3Umszs5SGOtQejIgBSWcBNwA9wIqIWCfpAmBtRKzclXJ36eGIiHgKuH9XrjUzK1InX3gTEauAVcP2ndfg3GNaKdNPpJlZpfgxYDOzhIZU7ttNTrpmVinp5lrZNU66ZlYpHRy9UAgnXTOrlE6NXihK4Ul30rg0eX3/SXvz4DO/SRIrpck9eySLtW3g6WSxUk0Y+ap1Iz40WYhpL1yaLNbsnqnJYvWxJVmsTvB0PYlUMeGaWfvcvWBmlpCHjJmZJTTolq6ZWTpu6ZqZJeSka2aWUMlnYHfSNbNqcUvXzCwhPwZsZpZQ2cfp5k1M+XJJ07L1yZI+JOl/JH1U0vQ0VTQza10HZwMuRN4caSuAp7L1i6lN3/PRbN/nC6yXmdkuKXvSzeteGBcROyegXBQRh2frP5Z0R6OL6mcDnj/jEPabMrvRqWZmHVX2dy/ktXTvlnRqtn7nzlkuJR0E7Gh0Uf1swE64ZpbSkFpfxkJe0j0deLWk+4AFwM2S+oDPZcfMzEplsI1lLOTNBrwVOEXSXsCB2fn9EfHrFJUzM2vXUMk7GFoaMhYRvwXuLLguZmaj5ocjzMwSKnc710nXzCrGLV0zs4QGVO62rpOumVVKuVOuk66ZVcxu372w8amHiw4BwMRxE5LESe30SQcni3XF9vuTxVr6xM1J4qScoXfzfd9MFmvpwr9OFqvbVGLImJlZtyh3ynXSNbOK2e27F8zMUhoseVs3790LZmZdpZOvdpS0RNIGSRslnTvC8XMkrZd0l6TvSnpBXplOumZWKdHGf81I6gEuAZZSe+HXSZIWDDvtdmqvvX0ZcA3wsbz6OemaWaV0sKW7GNgYEX0RsR24ClhWf0JEfD8idk70sBqYk1eok66ZVcoQ0fIiqVfS2rqlt66o2cCmuu3+bF8jpwG54wZ9I83MKqWd22gRsRxY3uDwSK85H7F4Se8AFgGvzovppGtmlTLQudEL/cDcuu05wObhJ0k6Dvgg8OqI+H1eoXmzAZ8taW6zc8zMyqRTN9KANcB8SfMkTQROBFbWnyBpIfBZ4PiI2NJK/fL6dD8M3CLpR5L+RtKsVgqt7yf53e8fa+USM7OO6NSNtGxS3rOAG4B7gKsjYp2kCyQdn53278BU4KuS7pC0skFxz8rrXugDjgCOA04APiTpNuBK4LpsRomRKvtsP8mcPzq03COVzaxSWmjBtl5WxCpg1bB959WtH9dumXkt3YiIoYi4MSJOA/YDPgUsoZaQzcxKpZMPRxQhr6X7nLt3EbGDWp/GSkmTC6uVmdkuGoxy/3Kdl3RPaHQgIp7ucF3MzEatq1/tGBE/T1URM7NO6GSfbhE8TtfMKsWvdjQzS6iruxfMzLqNuxfMzBLq9tELZmZdZbfvXnj/lMOKDgHARU+vSxIntU9uuzNZrH32mJEs1qtmDX8XdDFm90xNEgfSztD7zds/nSzWwYe8JVmsTvCNNDOzhNyna2aW0G7fvWBmllL4RpqZWTpln4LdSdfMKsXdC2ZmCbl7wcwsIbd0zcwS6uohY3WTsW2OiO9IejvwSmrzBS3PXmpuZlYa3f4Y8Oezc/aU9E5qE7BdBxwLLAbeWWz1zMza0+3dCy+NiJdJGg/8CtgvIgYlfRlo+HyqpF6gF+CEGYs5aur8jlXYzKyZsifdvIkpx2VdDHsBewLTs/17ABMaXRQRyyNiUUQscsI1s5QiouVlLOS1dC8D7gV6gA9Sm9u9DzgSuKrgupmZta3sLd28OdIukvRf2fpmSZcDxwGfi4hbU1TQzKwdXT16AWrJtm79CeCaQmtkZjYKg1Hulzt6nK6ZVYqfSDMzS6ir+3TNzLpN1/fpmpl1kyF3L5iZpeOWrplZQmUfvaCi7/QdOHNhkq+dDfdemyJMct02E6tVT8p/WxNmHqjRlnHQrEUt55yfP7K2aTxJS4CLqT0gdmlEfGTY8T2Ay4EjgN8AJ0TEA83KzHsM2Mysq0Qb/zUjqQe4BFgKLABOkrRg2GmnAY9HxIuAi4CP5tXPSdfMKmUoouUlx2JgY0T0RcR2aq8+WDbsnGXAF7P1a4BjJTVtPTvpmlmltNPSldQraW3d0ltX1GxgU912f7aPkc6JiAFgK7B3s/r5RpqZVcpgDLZ8bkQsB5Y3ODxSi3V487iVc57DSdfMKqWDgwP6gbl123OAzQ3O6c/eOz4deKxZoe5eMLNKGSJaXnKsAeZLmlc3ddnKYees5P9n0Hkr8L3Iyfpu6ZpZpXSqpRsRA5LOAm6gNmRsRUSsk3QBsDYiVlJ75/iXJG2k1sI9Ma9cJ10zq5ROPgYcEauAVcP2nVe3/gzwtnbKzE26kl4IvJlav8UA8AvgyojY2k4gM7MUyv4YcNM+XUlnA58BJgF/AkymlnxvlnRM4bUzM2vTYAy1vIyFvJbuGcBh2QzAFwKrIuIYSZ8Fvg4sHOmi+tmA954yh2mTZnayzmZmDZX9JeatjF7YmZj3oDYrMBHxIC3OBuyEa2YpdfCJtELktXQvBdZIWg0cTfZcsaRZ5IxFMzMbC2Vv6ebNBnyxpO8ALwYujIh7s/2PUEvCZmal0vXT9UTEOmBdgrqYmY1aV7d0zcy6TdlfYu6ka2aV4jnSzMwScveCmVlCZX8izUnXzCrFLV0zs4TK3qdLRJRyAXqrFMexuitWFT9TlWN101Lml5j35p/SVXEcq7tiVfEzVTlW1yhz0jUzqxwnXTOzhMqcdBvN0NmtcRyru2JV8TNVOVbXUNbhbWZmCZS5pWtmVjlOumZmCZUu6UpaImmDpI2Szi0wzgpJWyTdXVSMulhzJX1f0j2S1kl6T4GxJkm6VdKdWawPFRUri9cj6XZJ1xcc5wFJP5N0h6S1BceaIekaSfdmf2evKCjOwdnn2bk8Kem9BcV6X/bzcLekKyVNKiJOFus9WZx1RX2erjbWA4WHDabuAe4DDgQmAncCCwqKdTRwOHB3gs+1L3B4tr4X8PMCP5eAqdn6BOAW4MgCP9s5wFeA6wv+f/gAMLPov6ss1heB07P1icCMBDF7gIeBFxRQ9mzgfmBytn01cEpBn+NQ4G5gT2pPvH4HmJ/i761blrK1dBcDGyOiLyK2A1cBy4oIFBE/JNGUQxHxUET8NFv/LXAPtX8IRcSKiNiWbU7IlkLulkqaA7yR2rROlSBpGrUv5MsAImJ7RDyRIPSxwH0R8cuCyh8PTJY0nlpC3FxQnBcDqyPiqYgYAH4AvLmgWF2pbEl3NrCpbrufgpLTWJF0ALVZlG8pMEaPpDuALcC3I6KoWP8B/AOQ4q3RAdwo6bZstumiHAg8Anw+6za5VNKUAuPtdCJwZREFR8SvgI8DDwIPAVsj4sYiYlFr5R4taW9JewJvAOYWFKsrlS3paoR9lRnTJmkqcC3w3oh4sqg4ETEYEYcBc4DFkg7tdAxJbwK2RMRtnS67gaMi4nBgKXCmpKLm6BtPrdvp0xGxEPgdUNi9BQBJE4Hjga8WVP7zqP3GOA/YD5gi6R1FxIqIe6hNYPtt4FvUuggHiojVrcqWdPt57rfiHIr7NSgpSROoJdwrIuK6FDGzX4tvApYUUPxRwPGSHqDWDfRnkr5cQBwAImJz9ucW4GvUuqKK0A/01/12cA21JFykpcBPI+LXBZV/HHB/RDwSETuA64BXFhSLiLgsIg6PiKOpdeH9oqhY3ahsSXcNMF/SvOzb/0Rg5RjXadQkiVof4T0RcWHBsWZJmpGtT6b2D+7eTseJiA9ExJyIOIDa39P3IqKQ1pOkKZL22rkOvI7ar7EdFxEPA5skHZztOhZYX0SsOidRUNdC5kHgSEl7Zj+Lx1K7r1AISftkf+4P/AXFfrauU6r36UbEgKSzgBuo3c1dEbXZiDtO0pXAMcBMSf3A+RFxWRGxqLUK/wr4WdbXCvCPEbGqgFj7Al+U1EPtS/XqiCh0OFcCzwe+VssXjAe+EhHfKjDeu4Ersi/+PuDUogJl/Z6vBd5VVIyIuEXSNcBPqf2qfzvFPqJ7raS9gR3AmRHxeIGxuo4fAzYzS6hs3QtmZpXmpGtmlpCTrplZQk66ZmYJOemamSXkpGtmlpCTrplZQv8H+YKhproA3VkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Key:\n",
      "0 Mr . Woodhouse had so completely made up his mind to the visit , that in spite of the increasing coldness , he seemed to have no idea of shrinking from it , and set forward at last most punctually with his eldest daughter in his own carriage , with less apparent consciousness of the weather than either of the others ; too full of the wonder of his own going , and the pleasure it was to afford at Randalls to see that it was cold , and too well wrapt up to feel it .\n",
      "1 \" Oh !\n",
      "2 \" Oh no , no !\n",
      "3 Such was Jane Fairfax ' s history .\n",
      "4 \" That has been a good deal the case , my dear ; but not to the degree you mention .\n",
      "5 \" And I am quite serious too , I assure you ,\" replied Mrs . Elton gaily , \" in resolving to be always on the watch , and employing my friends to watch also , that nothing really unexceptionable may pass us .\"\n",
      "6 \" And here is Mrs . Weston and Mr . Frank Churchill too ! Quite delightful ; so many friends !\"\n",
      "7 \" You may well class the delight , the honour , and the comfort of such a situation together ,\" said Jane , \" they are pretty sure to be equal ; however , I am very serious in not wishing any thing to be attempted at present for me .\n",
      "8 Harriet , Mr . Elton , and Mr . Knightley , their own especial set , were the only persons invited to meet them ; the hours were to be early , as well as the numbers few ; Mr . Woodhouse ' s habits and inclination being consulted in every thing .\n",
      "9 \" Oh !\n"
     ]
    }
   ],
   "source": [
    "# Compute document similarity using LSA components\n",
    "similarity = np.asarray(np.asmatrix(X_test_lsa) * np.asmatrix(X_test_lsa).T)\n",
    "\n",
    "#Only taking the first 5 sentences\n",
    "sim_matrix=pd.DataFrame(similarity,index=X_test).iloc[0:10,0:10]\n",
    "\n",
    "#Making a plot\n",
    "ax = sns.heatmap(sim_matrix,yticklabels=range(10))\n",
    "plt.show()\n",
    "\n",
    "#Generating a key for the plot.\n",
    "print('Key:')\n",
    "for i in range(10):\n",
    "    print(i,sim_matrix.index[i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion and Discussion\n",
    "Question 0:  Does the test data identify similar sentences (as compared to the training data) for components 0 through 4 ?  \n",
    "\n",
    "Answer:  The test data does identify similar sentences with similar correlation scores.  However, the comonents do not match up exactly and some sentence are exclusively in training and test data. \n",
    "\n",
    "Question 1:  Change some hyperparameters a few times and see how that affects the results of the LSA.  \n",
    "\n",
    "Observations:  Changing the minimum word frequency from 2 to 4 reduced the number of features from 1948 to 1020.  It also created higher similarity scores for many of the elements in the components.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}