08-Classification/01-Classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datascience import *\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Regression\n",
    "\n",
    "A popular classification model is [logistic regression](https://en.wikipedia.org/wiki/Logistic_regression). This is what Underwood and Sellers use in their article to classify whether a text was reviewed or randomly selected from HathiTrust. Today we'll look at the difference between regression and classification tasks, and how we can use a logistic regression model to classify text like Underwood and Sellers. We won't have time to go through their full code, but if you're interested I've provided a walk-through in the second notebook.\n",
    "\n",
    "To explore the regression model let's first create some dummy data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_tb = Table()\n",
    "demo_tb['Study_Hours'] = [2.0, 6.9, 1.6, 7.8, 3.1, 5.8, 3.4, 8.5, 6.7, 1.6, 8.6, 3.4, 9.4, 5.6, 9.6, 3.2, 3.5, 5.9, 9.7, 6.5]\n",
    "demo_tb['Grade'] = [67.0, 83.6, 35.4, 79.2, 42.4, 98.2, 67.6, 84.0, 93.8, 64.4, 100.0, 61.6, 100.0, 98.4, 98.4, 41.8, 72.0, 48.6, 90.8, 100.0]\n",
    "demo_tb['Pass'] = [0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1]\n",
    "demo_tb.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intuiting the Linear Regression Model\n",
    "\n",
    "You may have encountered linear regression in previous coursework of yours. Linear regression, in its simple form, tries to model the relationship between two continous variables as a straight line. It interprets one variable as the input, and the other as the output.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_tb.scatter('Study_Hours','Grade')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, we're interested in `Study_Hours` and `Grade`. This is a natural \"input\" \"output\" situation. To plot the regression line, or ***best-fit***, we can feed in `fit_line=True` to the `scatter` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_tb.scatter('Study_Hours','Grade', fit_line=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The better this line fits the points, the better we can predict one's `Grade` based on their `Study_Hours`, even if we've never seen anyone put in that number of study hours before.\n",
    "\n",
    "The regression model above can be expressed as:\n",
    "\n",
    "$GRADE_i= \\alpha + \\beta STUDYHOURS + \\epsilon_i$\n",
    "\n",
    "The variable we want to predict (or model) is the left side `y` variable, here `GRADE`. The variable which we think has an influence on our left side variable is on the right side, the independent variable `STUDYHOURS`. The $\\alpha$ term is the y-intercept and the $\\epsilon_i$ describes the randomness.\n",
    "\n",
    "The $\\beta$ coefficient on `STUDYHOURS` gives us the slope, in a univariate regression. That's the factor on `STUDYHOURS` to get `GRADE`.\n",
    "\n",
    "If we want to build a model for the regression, we can use the `sklearn` library. `sklearn` is by far the most popular machine learning library for Python, and its syntax is really important to learn. In the next cell we'll import the [`Linear Regression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model and assign it to a `linreg` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "linreg = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go any further, `sklearn` likes our data in a very specific format. The `X` must be in an array of arrays, each sub array is an observation. Because we only have one independent variable, we'll have sub arrays of `len` 1. We can do that with the `reshape` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = demo_tb['Study_Hours'].reshape(-1,1)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your output, or dependent variable, is just one array with no sub arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = demo_tb['Grade'].reshape(len(demo_tb['Grade']),)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use the `fit` method to fit the model. This happens in-place, so we don't have to reassign the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linreg.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get back the `intercept_` and $\\beta$ `coef_` with attributes of the `linreg` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B0, B1 = linreg.intercept_, linreg.coef_[0]\n",
    "B0, B1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this means:\n",
    "\n",
    "$GRADE_i= 42.897229302892598 + 5.9331153718275509 * STUDYHOURS + \\epsilon_i$\n",
    "\n",
    "As a linear regression this is simple to interpret. To get our grade score, we take the number of study hours and multipy it by 5.9331153718275509 then we add 42.897229302892598 and that's our prediction.\n",
    "\n",
    "If we look at our chart again but using the model we just made, that looks about right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = linreg.predict(X)\n",
    "print(X)\n",
    "print(y_pred)\n",
    "plt.scatter(X, y)\n",
    "plt.plot(X, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate how great our model is with the `score` method. We need to give it the `X` and observed `y` values, and it will predict its own `y` values and compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linreg.score(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Linear Regression, `sklearn` returns an **R-squared** from the `score` method. The R-squared tells us how much of the variation in the data can be explained by our model, .559 isn't that bad, but obviously more goes into your `Grade` than *just* `Study_Hours`.\n",
    "\n",
    "Nevertheless we can still predict a grade just like we did above to create that line, let's say I studied for 5 hours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linreg.predict([[5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe I should study more?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linreg.predict([[20]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! I rocked it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intuiting the Logistic Regression Model\n",
    "\n",
    "But what happens if one of your variables is categorical, and not continuous? Suppose we don't care about the `Grade` score, but we just care if you `Pass` or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_tb.scatter('Study_Hours','Pass')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How would we fit a line to that? That's where the [logistic function](https://en.wikipedia.org/wiki/Logistic_function) can be handy. The general logistic function is:\n",
    "\n",
    "$ f(x) = \\frac{1}{1 + e^{-x}} $\n",
    "\n",
    "We can translate that to Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic(p):\n",
    "    return 1 / (1 + np.exp(-p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also need to assign a couple $\\beta$ coefficients for the intercept and variable just like we saw in linear regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B0, B1 = 0, 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the logistic curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmin, xmax = -10,10\n",
    "xlist = [float(x)/int(1e4) for x in range(xmin*int(1e4), xmax*int(1e4))]  # just a lot of points on the x-axis\n",
    "ylist = [logistic(B0 + B1*x) for x in xlist]\n",
    "\n",
    "plt.axis([-10, 10, -0.1,1.1])\n",
    "plt.plot(xlist,ylist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When things get complicated, however, with several independent variables, we don't want to write our own code. Someone has done that for us. We'll go back to `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll `reshape` our arrays again too, since we know how `sklearn` likes them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = demo_tb['Study_Hours'].reshape(-1,1)\n",
    "y = demo_tb['Pass'].reshape(len(demo_tb['Pass']),)\n",
    "X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `fit` function again on our `X` and `y`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get those $\\beta$ coefficients back out from `sklearn` for our grade data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B0, B1 = lr.intercept_[0], lr.coef_[0][0]\n",
    "B0, B1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can plot the curve just like we did earlier, and we'll add our points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmin, xmax = 0,10\n",
    "xlist = [float(x)/int(1e4) for x in range(xmin*int(1e4), xmax*int(1e4))]\n",
    "ylist = [logistic(B0 + B1*x) for x in xlist]\n",
    "plt.plot(xlist,ylist)\n",
    "\n",
    "# add our \"observed\" data points\n",
    "plt.scatter(demo_tb['Study_Hours'],demo_tb['Pass'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How might this curve be used for a binary classification task?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Classification\n",
    "\n",
    "That's great, so we can begin to see how we might use such a model to conduct binary classification. In this task, we want to get a number of study hours as an observation, and place it in one of two bins: pass or fail.\n",
    "\n",
    "To create the model though, we have to train it on the data we have. [In machine learning, we also need to put some data aside as \"testing data\" so that we don't bias our model by using it in the training process.](https://en.wikipedia.org/wiki/Training,_test,_and_validation_sets) In Python we often see `X_train`, `y_train` and `X_test`, `y_test`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = demo_tb.column('Study_Hours')[:-2]\n",
    "y_train = demo_tb.column('Pass')[:-2]\n",
    "\n",
    "X_test = demo_tb.column('Study_Hours')[-2:]\n",
    "y_test = demo_tb.column('Pass')[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the observations we're setting aside for later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll fit our model again but only on the `_train` data, and get out the $\\beta$ coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.fit(X_train.reshape(-1,1),y_train.reshape(len(y_train),))\n",
    "B0, B1 = lr.intercept_[0], lr.coef_[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can send these coefficients back into the `logistic` function we wrote earlier to get the probability that a student would pass given our `X_test` values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fitted = [logistic(B1*th + B0) for th in X_test]\n",
    "fitted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take the probability and change this to a binary outcome based on probability `>` or `<` .5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = [pred >.5 for pred in fitted]\n",
    "prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sklearn` built-in methods can make this `predict` process faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.predict(X_test.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how accurate our model is, we'd predict on the \"unseen\" `_test`ing data and see how many we got correct. In this case there's only two, so not a whole lot to test with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_eval = [prediction[i]==y_test[i] for i in range(len(prediction))]\n",
    "float(sum(prediction_eval)/len(prediction_eval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do this quickly in `sklearn` too with the `score` method like in the linear regression example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.score(X_test.reshape(-1, 1), y_test.reshape(len(y_test),))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Classification of Textual Data\n",
    "\n",
    "How can we translate this simple model of binary classification to text? I'm going to leave the more complicated model that Underwood and Sellers use for the next notebook if you're interested, today we're just going to work through the basic classfication pipeline. We'll download a pre-made corpus from `nltk`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download(\"movie_reviews\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we import the `movie_reviews` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might expect, this is a corpus of IMDB movie reviews. Someone went through and read each review, labeling it as either \"positive\" or \"negative\". The task we have before us is to create a model that can accurately predict whether a never-before-seen review is positive or negative. This is analogous to Underwood and Sellers looking at whether a poem volume was reviewed or randomly selected.\n",
    "\n",
    "From the `movie_reviews` object let's take out the reviews and the judgement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = [movie_reviews.raw(fileid) for fileid in movie_reviews.fileids()]\n",
    "judgements = [movie_reviews.categories(fileid)[0] for fileid in movie_reviews.fileids()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the first review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(reviews[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you consider this a positive or negative review? Let's see what the human annotator said:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(judgements[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So right now we have a list of movie reviews in the `reviews` variable and a list of their corresponding judgements in the `judgements` variable. Awesome. What does this sound like to you? Independent and dependent variables? You'd be right!\n",
    "\n",
    "`reviews` is our `X` array from above. `judgements` is our `y` array from above. Let's first reassign our `X` and `y` so we're explicit about what's going on. While we're at it, we're going to set the random `seed` for our computer. This just makes our result reproducible. We'll also `shuffle` so that we randomize the order of our observations, and when we split the testing and training data it won't be in a biased order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "np.random.seed(1)\n",
    "\n",
    "X, y = shuffle(reviews, judgements, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't believe me that all we did is reassign and shuffle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[0], y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get meaningful independent variables (words) we have to do some processing too (think DTM!). With `sklearn`'s text pipelines, we can quickly build a text a classifier in only a few lines of Python: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LogisticRegression?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score, train_test_split\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))),\n",
    "                    ('tfidf', TfidfTransformer()),\n",
    "                    ('clf', LogisticRegression(random_state=0, penalty='l2', C=1000))\n",
    "                     ])\n",
    "\n",
    "scores = cross_val_score(text_clf, X, y, cv=5)\n",
    "\n",
    "print(scores, np.mean(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Whoa! What just happened?!?*** The pipeline tells us three things happened:\n",
    "\n",
    "1. `CountVectorizer`\n",
    "\n",
    "2. `TfidfTransformer`\n",
    "\n",
    "3. `LogisticRegression`\n",
    "\n",
    "Let's walk through this step by step.\n",
    "\n",
    "1. A count vectorizer does exactly what we did last week with tokenization. It changes all the texts to words, and then simply counts the frequency of each word occuring in the corpus for each document. The feature array for each document at this point is simply the length of all unique words in a corpus, with the count for the frequency of each. This is the most basic way to provide features for a classifier---a document term matrix.\n",
    "\n",
    "2. [tfidf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) (term frequency inverse document frequency) is an algorithm that aims to find words that are important to specific documents. It does this by taking the term frequency (tf) for a specific term in a specific document, and multiplying it by the term's inverse document frequency (idf), which is the total number of documents divided by the number of documents that contain the term at least once. Thus, idf is defined as:\n",
    "\n",
    "$$idf(t, d, D)= log\\left(\\frac{\\mid D \\mid}{\\mid \\{d \\subset D : t \\subset d \\} \\mid}\\right )$$\n",
    "\n",
    "So tfidf is simply:\n",
    "\n",
    "$$tfidf(t, d, D)= f_{t,d}*log\\left(\\frac{\\mid D \\mid}{\\mid \\{d \\subset D : t \\subset d \\} \\mid}\\right )$$\n",
    "\n",
    "A tfidf value is calculated for each term for each document. The feature arrays for a document is now the tfidf values. ***The tfidf matrix is the exact same as our document term matrix, only now the values have been weighted according to their distribution across documents.***\n",
    "\n",
    "The pipeline now sends these tfidf feature arrays to a 3. **Logistic Regression**, what we learned above. We add in an l2 penalization parameter because we have many more independent variables from our `dtm` than observations. The independent variables are the tfidf values of each word. In a simple linear model, that would look like:\n",
    "\n",
    "$$log(CLASSIFICATION_i)= \\alpha + \\beta DOG + \\beta RABBIT + \\beta JUMP + ... + \\epsilon_i$$\n",
    "\n",
    "where $\\beta DOG$ is the model's $\\beta$ coefficient multiplied by the ***tfidf*** value for \"dog\".\n",
    "\n",
    "The code below breaks this down by each step, but combines the `CountVectorizer` and `TfidfTransformer` in the `TfidfVectorizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=50)\n",
    "\n",
    "# get tfidf values\n",
    "tfidf = TfidfVectorizer()\n",
    "tfidf.fit(X)\n",
    "X_train = tfidf.transform(X_train)\n",
    "X_test = tfidf.transform(X_test)\n",
    "\n",
    "# build and test logit\n",
    "logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)\n",
    "model = logit_class.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The concise code we first ran actually uses \"cross validation\", where we split up testing and training data `k` number of times and average our score on all of them. This is a more reliable metric than just testing the accuracy once. It's possible that you're random train/test split just didn't provide a good split, so averaging it over multiple splits is preferred.\n",
    "\n",
    "You'll also notice the `ngram_range` parameter in the `CountVectorizer` in the first cell. This expands our vocabulary document term matrix by including groups of words together. It's easier to understand an [ngram](https://en.wikipedia.org/wiki/N-gram) by just seeing one. We'll look at a bigram (bi is for 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.util import ngrams\n",
    "\n",
    "ngs = ngrams(\"Text analysis is so cool. I can really see why classification can be a valuable tool.\".split(), 2)\n",
    "list(ngs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trigram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ngs = ngrams(\"Text analysis is so cool. I can really see why classification can be a valuable tool.\".split(), 3)\n",
    "list(ngs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You get the point. This helps us combat this \"bag of words\" idea, but doesn't completely save us. For our purposes here, just as we counted the frequency of individual words, we've added counting the frequency of groups of 2s and 3s.\n",
    "\n",
    "---\n",
    "\n",
    "### Important Features\n",
    "\n",
    "After we train the model we can then index the tfidf matrix for the words with the most significant coefficients (remember independent variables!) to get the most helpful features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = tfidf.get_feature_names()\n",
    "top10pos = np.argsort(model.coef_[0])[-10:]\n",
    "print(\"Top features for positive reviews:\")\n",
    "print(list(feature_names[j] for j in top10pos))\n",
    "print()\n",
    "print(\"Top features for negative reviews:\")\n",
    "top10neg = np.argsort(model.coef_[0])[:10]\n",
    "print(list(feature_names[j] for j in top10neg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "We can also use our model to classify new reviews, all we have to do is extract the tfidf features from the raw text and send them to the model as our features (independent variables):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_bad_review = \"This movie really sucked. I can't believe how long it dragged on. The actors are absolutely terrible. They should rethink their career paths\"\n",
    "\n",
    "features = tfidf.transform([new_bad_review])\n",
    "\n",
    "model.predict(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_good_review = \"I loved this film! The cinematography was incredible, and Leonardo Dicarpio is flawless. Super cute BTW.\"\n",
    "\n",
    "features = tfidf.transform([new_good_review])\n",
    "\n",
    "model.predict(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework\n",
    "\n",
    "Let's examine more the three objects in the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CountVectorizer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TfidfTransformer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LogisticRegression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've copied the cell from above below. Try playing with the parameters to these objects and see if you can improve the `cross_val_score` for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))),\n",
    "                    ('tfidf', TfidfTransformer()),\n",
    "                    ('clf', LogisticRegression(random_state=0))\n",
    "                     ])\n",
    "\n",
    "scores = cross_val_score(text_clf, X, y, cv=5)\n",
    "\n",
    "print(scores, np.mean(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why do you think your score improved (or didn't)?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# BONUS (not assigned)\n",
    "\n",
    "We're going to download the [20 Newsgroups](http://qwone.com/~jason/20Newsgroups/), a widely used corpus for demos of general texts:\n",
    "\n",
    "> The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll import the data from `sklearn`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what categories they have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fetch_20newsgroups(subset=\"train\").target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `subset` parameter will give you training and testing data. You can also use the `categories` parameter to choose only certain categories.\n",
    "\n",
    "If we wanted to get the training data for `sci.electronics` and `rec.autos` we would write this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = fetch_20newsgroups(subset=\"train\", categories=['sci.electronics', 'rec.autos'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list of documents (strings) is in the `.data` property, we can access the first one like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is the assigment category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.target[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many training documents are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(train.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same for the testing data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = fetch_20newsgroups(subset=\"test\", categories=['sci.electronics', 'rec.autos'])\n",
    "test.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.target[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(test.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now have your four lists:\n",
    "\n",
    "- `train.data` = `X_train` list of strings\n",
    "- `train.target` = `y_train` list of category assignments\n",
    "- `test.data` = `X_test` list of strings\n",
    "- `test.target` = `y_test` list of category assignments\n",
    "\n",
    "Build a classifier below. And then choose different categories and rebuild it. Which categories are easier to classify from each other? Why?\n",
    "\n",
    "***TIP***: Don't forget to `shuffle` your data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}