4.4.2_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction \n",
    "This notebook imports the popular stories 'Alice in Wonderland' and 'Persuasion' and tries to predict if any given sentence is from \"Alice\" or \"Persuasion\".  \n",
    "\n",
    "The first step is to import the necessary modules and import the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy\n",
    "import sklearn\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import re\n",
    "from nltk.corpus import gutenberg, stopwords\n",
    "from collections import Counter\n",
    "\n",
    "import nltk\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the nlp library, spacy\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package gutenberg to\n",
      "[nltk_data]     C:\\Users\\Fred\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package gutenberg is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('gutenberg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in c:\\users\\fred\\anaconda3\\lib\\site-packages (2.0.0)\n",
      "\n",
      "    Linking successful\n",
      "    C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\en_core_web_sm -->\n",
      "    C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\spacy\\data\\en\n",
      "\n",
      "    You can now load the model via spacy.load('en')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# this takes a long time\n",
    "\n",
    "### need to run as administrator from Anaconda3 promplt\n",
    "\n",
    "!python -m spacy download en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clean and explore the data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility function for standard text cleaning.\n",
    "def text_cleaner(text):\n",
    "    # Visual inspection identifies a form of punctuation spaCy does not\n",
    "    # recognize: the double dash '--'.  Better get rid of it now!\n",
    "    text = re.sub(r'--',' ',text)\n",
    "    text = re.sub(\"[\\[].*?[\\]]\", \"\", text)\n",
    "    text = ' '.join(text.split())\n",
    "    return text\n",
    "    \n",
    "# Load and clean the data.\n",
    "persuasion = gutenberg.raw('austen-persuasion.txt')\n",
    "alice = gutenberg.raw('carroll-alice.txt')\n",
    "\n",
    "# The Chapter indicator is idiosyncratic\n",
    "persuasion = re.sub(r'Chapter \\d+', '', persuasion)\n",
    "alice = re.sub(r'CHAPTER .*', '', alice)\n",
    "    \n",
    "alice = text_cleaner(alice[:int(len(alice)/10)])\n",
    "persuasion = text_cleaner(persuasion[:int(len(persuasion)/10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parse the cleaned novels. This can take a bit.\n",
    "# had to do this in an admin terminal.......\n",
    "nlp = spacy.load('en')\n",
    "alice_doc = nlp(alice)\n",
    "persuasion_doc = nlp(persuasion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(Alice, was, beginning, to, get, very, tired, of, sittin...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(So, she, was, considering, in, her, own, mind, (, as, w...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(There, was, nothing, so, VERY, remarkable, in, that, ;,...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(Oh, dear, !)</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(I, shall, be, late, !, ')</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                             0        1\n",
       "0  (Alice, was, beginning, to, get, very, tired, of, sittin...  Carroll\n",
       "1  (So, she, was, considering, in, her, own, mind, (, as, w...  Carroll\n",
       "2  (There, was, nothing, so, VERY, remarkable, in, that, ;,...  Carroll\n",
       "3                                                (Oh, dear, !)  Carroll\n",
       "4                                   (I, shall, be, late, !, ')  Carroll"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Group into sentences.\n",
    "alice_sents = [[sent, \"Carroll\"] for sent in alice_doc.sents]\n",
    "persuasion_sents = [[sent, \"Austen\"] for sent in persuasion_doc.sents]\n",
    "\n",
    "# Combine the sentences from the two novels into one data frame.\n",
    "sentences = pd.DataFrame(alice_sents + persuasion_sents)\n",
    "\n",
    "pd.set_option('max_colwidth', 60)\n",
    "\n",
    "sentences.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility function to create a list of the 2000 most common words.\n",
    "def bag_of_words(text):\n",
    "    \n",
    "    # Filter out punctuation and stop words.\n",
    "    allwords = [token.lemma_\n",
    "                for token in text\n",
    "                if not token.is_punct\n",
    "                and not token.is_stop]\n",
    "    \n",
    "    # Return the most common words.\n",
    "    return [item[0] for item in Counter(allwords).most_common(2000)]\n",
    "    \n",
    "\n",
    "# Creates a data frame with features for each word in our common word set.\n",
    "# Each value is the count of the times the word appears in each sentence.\n",
    "def bow_features(sentences, common_words):\n",
    "    \n",
    "    # Scaffold the data frame and initialize counts to zero.\n",
    "    df = pd.DataFrame(columns=common_words)\n",
    "    df['text_sentence'] = sentences[0]\n",
    "    df['text_source'] = sentences[1]\n",
    "    df.loc[:, common_words] = 0\n",
    "    \n",
    "    # Process each row, counting the occurrence of words in each sentence.\n",
    "    for i, sentence in enumerate(df['text_sentence']):\n",
    "        \n",
    "        # Convert the sentence to lemmas, then filter out punctuation,\n",
    "        # stop words, and uncommon words.\n",
    "        words = [token.lemma_\n",
    "                 for token in sentence\n",
    "                 if (\n",
    "                     not token.is_punct\n",
    "                     and not token.is_stop\n",
    "                     and token.lemma_ in common_words\n",
    "                 )]\n",
    "        \n",
    "        # Populate the row with word counts.\n",
    "        for word in words:\n",
    "            df.loc[i, word] += 1\n",
    "        \n",
    "        # This counter is just to make sure the kernel didn't hang.\n",
    "        if i % 50 == 0:\n",
    "            print(\"Processing row {}\".format(i))\n",
    "            \n",
    "    return df\n",
    "\n",
    "# Set up the bags.\n",
    "alicewords = bag_of_words(alice_doc)\n",
    "persuasionwords = bag_of_words(persuasion_doc)\n",
    "\n",
    "# Combine bags to create a set of unique words.\n",
    "common_words = set(alicewords + persuasionwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing row 0\n",
      "Processing row 50\n",
      "Processing row 100\n",
      "Processing row 150\n",
      "Processing row 200\n",
      "Processing row 250\n",
      "Processing row 300\n",
      "Processing row 350\n",
      "Processing row 400\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>disappoint</th>\n",
       "      <th>nice</th>\n",
       "      <th>seat</th>\n",
       "      <th>range</th>\n",
       "      <th>continue</th>\n",
       "      <th>anybody</th>\n",
       "      <th>torment</th>\n",
       "      <th>plan</th>\n",
       "      <th>eagerly</th>\n",
       "      <th>worst</th>\n",
       "      <th>...</th>\n",
       "      <th>dream</th>\n",
       "      <th>yes</th>\n",
       "      <th>honourable</th>\n",
       "      <th>lawyer</th>\n",
       "      <th>art</th>\n",
       "      <th>turkey</th>\n",
       "      <th>cheek</th>\n",
       "      <th>earnestly</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Alice, was, beginning, to, get, very, tired, of, sittin...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(So, she, was, considering, in, her, own, mind, (, as, w...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(There, was, nothing, so, VERY, remarkable, in, that, ;,...</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Oh, dear, !)</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(I, shall, be, late, !, ')</td>\n",
       "      <td>Carroll</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1649 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  disappoint nice seat range continue anybody torment plan eagerly worst  ...  \\\n",
       "0          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "1          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "2          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "3          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "4          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "\n",
       "  dream yes honourable lawyer art turkey cheek earnestly  \\\n",
       "0     0   0          0      0   0      0     0         0   \n",
       "1     0   0          0      0   0      0     0         0   \n",
       "2     0   0          0      0   0      0     0         0   \n",
       "3     0   0          0      0   0      0     0         0   \n",
       "4     0   0          0      0   0      0     0         0   \n",
       "\n",
       "                                                 text_sentence text_source  \n",
       "0  (Alice, was, beginning, to, get, very, tired, of, sittin...     Carroll  \n",
       "1  (So, she, was, considering, in, her, own, mind, (, as, w...     Carroll  \n",
       "2  (There, was, nothing, so, VERY, remarkable, in, that, ;,...     Carroll  \n",
       "3                                                (Oh, dear, !)     Carroll  \n",
       "4                                   (I, shall, be, late, !, ')     Carroll  \n",
       "\n",
       "[5 rows x 1649 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create our data frame with features. This can take a while to run.\n",
    "word_counts = bow_features(sentences, common_words)\n",
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>disappoint</th>\n",
       "      <th>nice</th>\n",
       "      <th>seat</th>\n",
       "      <th>range</th>\n",
       "      <th>continue</th>\n",
       "      <th>anybody</th>\n",
       "      <th>torment</th>\n",
       "      <th>plan</th>\n",
       "      <th>eagerly</th>\n",
       "      <th>worst</th>\n",
       "      <th>...</th>\n",
       "      <th>yes</th>\n",
       "      <th>honourable</th>\n",
       "      <th>lawyer</th>\n",
       "      <th>art</th>\n",
       "      <th>turkey</th>\n",
       "      <th>cheek</th>\n",
       "      <th>earnestly</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "      <th>num_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Alice, was, beginning, to, get, very, tired, of, sittin...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(So, she, was, considering, in, her, own, mind, (, as, w...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(There, was, nothing, so, VERY, remarkable, in, that, ;,...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Oh, dear, !)</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(I, shall, be, late, !, ')</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1650 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  disappoint nice seat range continue anybody torment plan eagerly worst  ...  \\\n",
       "0          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "1          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "2          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "3          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "4          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "\n",
       "  yes honourable lawyer art turkey cheek earnestly  \\\n",
       "0   0          0      0   0      0     0         0   \n",
       "1   0          0      0   0      0     0         0   \n",
       "2   0          0      0   0      0     0         0   \n",
       "3   0          0      0   0      0     0         0   \n",
       "4   0          0      0   0      0     0         0   \n",
       "\n",
       "                                                 text_sentence text_source  \\\n",
       "0  (Alice, was, beginning, to, get, very, tired, of, sittin...     Carroll   \n",
       "1  (So, she, was, considering, in, her, own, mind, (, as, w...     Carroll   \n",
       "2  (There, was, nothing, so, VERY, remarkable, in, that, ;,...     Carroll   \n",
       "3                                                (Oh, dear, !)     Carroll   \n",
       "4                                   (I, shall, be, late, !, ')     Carroll   \n",
       "\n",
       "  num_words  \n",
       "0        67  \n",
       "1        63  \n",
       "2        33  \n",
       "3         3  \n",
       "4         6  \n",
       "\n",
       "[5 rows x 1650 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts['num_words'] = word_counts['text_sentence'].apply(lambda x: len(x))\n",
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions count the number of adverbs, interjections and pronouns and adds the columns to the word_counts dataframe in hopes that these additional features will improve the accuracy of the 4 models:  Random Forest, Logistic Regression, Gradient Boosting, and Support Vector Machines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the adverbs and add the column to the word_counts dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"RB\" else 0).sum()\n",
    "#        count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_adverbs'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# -------------------------------------------------------------------------------------------------------\n",
    "# Count the interjections and add the column to the dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"UH\" else 0).sum()\n",
    "#        count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_inter'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# ------------------------------------------------------------------------------------------------------\n",
    "# Count the pronouns and add the column to the dataframe\n",
    "\n",
    "def count_verbs(txt):\n",
    "    sentences = nltk.sent_tokenize(str(txt))\n",
    "    count = 0\n",
    "    for sentence in sentences:    \n",
    "        text = nltk.word_tokenize(sentence)\n",
    "        tag = nltk.pos_tag(text)\n",
    "        a = pd.Series(tag)\n",
    "        a = a.map(lambda x: 1 if x[1] == \"PRP\" else 0).sum()\n",
    "#        count = count + a\n",
    "    return a\n",
    "\n",
    "word_counts['count_pronoun'] = word_counts['text_sentence'].apply(lambda x: count_verbs(x))\n",
    "\n",
    "# ------------------------------------------------------------------------------------------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the first 5 lines of the dataframe to make sure the knew features are captured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>disappoint</th>\n",
       "      <th>nice</th>\n",
       "      <th>seat</th>\n",
       "      <th>range</th>\n",
       "      <th>continue</th>\n",
       "      <th>anybody</th>\n",
       "      <th>torment</th>\n",
       "      <th>plan</th>\n",
       "      <th>eagerly</th>\n",
       "      <th>worst</th>\n",
       "      <th>...</th>\n",
       "      <th>art</th>\n",
       "      <th>turkey</th>\n",
       "      <th>cheek</th>\n",
       "      <th>earnestly</th>\n",
       "      <th>text_sentence</th>\n",
       "      <th>text_source</th>\n",
       "      <th>num_words</th>\n",
       "      <th>count_adverbs</th>\n",
       "      <th>count_inter</th>\n",
       "      <th>count_pronoun</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Alice, was, beginning, to, get, very, tired, of, sittin...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>67</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(So, she, was, considering, in, her, own, mind, (, as, w...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>63</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(There, was, nothing, so, VERY, remarkable, in, that, ;,...</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>33</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(Oh, dear, !)</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>(I, shall, be, late, !, ')</td>\n",
       "      <td>Carroll</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1653 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  disappoint nice seat range continue anybody torment plan eagerly worst  ...  \\\n",
       "0          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "1          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "2          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "3          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "4          0    0    0     0        0       0       0    0       0     0  ...   \n",
       "\n",
       "  art turkey cheek earnestly  \\\n",
       "0   0      0     0         0   \n",
       "1   0      0     0         0   \n",
       "2   0      0     0         0   \n",
       "3   0      0     0         0   \n",
       "4   0      0     0         0   \n",
       "\n",
       "                                                 text_sentence text_source  \\\n",
       "0  (Alice, was, beginning, to, get, very, tired, of, sittin...     Carroll   \n",
       "1  (So, she, was, considering, in, her, own, mind, (, as, w...     Carroll   \n",
       "2  (There, was, nothing, so, VERY, remarkable, in, that, ;,...     Carroll   \n",
       "3                                                (Oh, dear, !)     Carroll   \n",
       "4                                   (I, shall, be, late, !, ')     Carroll   \n",
       "\n",
       "  num_words count_adverbs count_inter count_pronoun  \n",
       "0        67             2           0             3  \n",
       "1        63             5           0             3  \n",
       "2        33             4           0             2  \n",
       "3         3             0           1             0  \n",
       "4         6             1           0             1  \n",
       "\n",
       "[5 rows x 1653 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_counts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the models  \n",
    "The \"text_source\" is what the models are trying to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.984\n",
      "\n",
      "Test set score: 0.8083832335329342\n"
     ]
    }
   ],
   "source": [
    "from sklearn import ensemble\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "rfc = ensemble.RandomForestClassifier()\n",
    "Y = word_counts['text_source']\n",
    "X = np.array(word_counts.drop(['text_sentence','text_source'], 1))\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    Y,\n",
    "                                                    test_size=0.4,\n",
    "                                                    random_state=0)\n",
    "# normalize the training data\n",
    "X_train = sklearn.preprocessing.normalize(X_train)\n",
    "\n",
    "# normalize the test data\n",
    "X_test = sklearn.preprocessing.normalize(X_test)\n",
    "\n",
    "train = rfc.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', rfc.score(X_train, y_train))\n",
    "print('\\nTest set score:', rfc.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Fred\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(250, 1651) (250,)\n",
      "Training set scorae: 0.688\n",
      "\n",
      "Test set score: 0.6826347305389222\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lr = LogisticRegression(penalty='l2') # No need to specify l2 as it's the default. But we put it for demonstration.\n",
    "train = lr.fit(X_train, y_train)\n",
    "print(X_train.shape, y_train.shape)\n",
    "print('Training set scorae:', lr.score(X_train, y_train))\n",
    "print('\\nTest set score:', lr.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.996\n",
      "\n",
      "Test set score: 0.8323353293413174\n"
     ]
    }
   ],
   "source": [
    "clf = ensemble.GradientBoostingClassifier()\n",
    "train = clf.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', clf.score(X_train, y_train))\n",
    "print('\\nTest set score:', clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.688\n",
      "\n",
      "Test set score: 0.6826347305389222\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "sv_c = SVC(gamma='auto')\n",
    "train = sv_c.fit(X_train, y_train)\n",
    "\n",
    "print('Training set score:', sv_c.score(X_train, y_train))\n",
    "print('\\nTest set score:', sv_c.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation and Conclusion  \n",
    "\n",
    "The final accuracy scores (for test data set) for the 4 models are as follows:  \n",
    "\n",
    "Random Forest:  0.81  \n",
    "Logistic Regression:  0.68  \n",
    "Gradient Boosting:  0.83  \n",
    "Support Vector Machines:  0.68  \n",
    "\n",
    "Unfortunately these values decreased when normalizing the data (which was performed above).  The values also did not significantly improve when adding the features:     \n",
    "\n",
    "- the number of words in each sentence  \n",
    "- the number of adverbs\n",
    "- the number of interjections  \n",
    "- the number of pronouns  \n",
    "\n",
    "Additional feature engineering could be performed to improve these results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}