11-Word-Embeddings/01-Word-Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')\n",
    "from datascience import *\n",
    "import numpy as np\n",
    "from scipy.spatial.distance import cosine\n",
    "import gensim\n",
    "import nltk\n",
    "from string import punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embedding\n",
    "\n",
    "This lesson is designed to explore features of word embeddings described by Ben Schmidt in his blog post <a href=\"http://bookworm.benschmidt.org/posts/2015-10-30-rejecting-the-gender-binary.html\">\"Rejecting the Gender Binary\"</a>.\n",
    "\n",
    "The primary corpus we use consists of the <a href=\"http://txtlab.org/?p=601\">150 English-language novels</a> made available by the .txtLab at McGill. We also look at a <a href=\"http://ryanheuser.org/word-vectors-1/\">Word2Vec model trained on the ECCO-TCP corpus</a> of 2,350 eighteenth-century literary texts made available by Ryan Heuser. (I have shortened the number of terms in the model by half in order to conserve memory.)\n",
    "\n",
    "For background on Word2Vec's mechanics, I suggest this <a href=\"https://www.tensorflow.org/versions/r0.8/tutorials/word2vec/index.html\">brief tutorial</a> by Google, especially the sections \"Motivation,\" \"Skip-Gram Model,\" and \"Visualizing.\"\n",
    "\n",
    "We'll read in Andrew Piper's corpus we used in our Topic Modeling notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_tb = Table.read_table('../09-Topic-Modeling/data/txtlab_Novel150_English.csv')\n",
    "\n",
    "fiction_path = '../09-Topic-Modeling/data/txtlab_Novel150_English/'\n",
    "\n",
    "novel_list = []\n",
    "\n",
    "# Iterate through filenames in metadata table\n",
    "for filename in metadata_tb['filename']:\n",
    "    \n",
    "    # Read in novel text as single string, make lowercase\n",
    "    with open(fiction_path + filename, 'r') as file_in:\n",
    "        novel = file_in.read()\n",
    "    \n",
    "    # Add novel text as single string to master list\n",
    "    novel_list.append(novel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Pre-Processing\n",
    "\n",
    "Word2Vec learns about the relationships among words by observing them in context. We'll need to tokenize the words in our corpus while retaining sentence boundaries. Since novels were imported as single strings, we'll first use <i>sent_tokenize</i> to divide them into sentences, and second, we'll split each sentence into its own list of words.\n",
    "\n",
    "We'll use `nltk`'s sentence tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to memory and time constraints we'll use our quick and dirty tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_tokenize(text):\n",
    "    \n",
    "    # Iterate through text removing punctuation characters\n",
    "    no_punct = \"\".join([char for char in text if char not in punctuation])\n",
    "    \n",
    "    # Split text over whitespace into list of words\n",
    "    tokens = no_punct.split()\n",
    "    \n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First get the sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [sentence for novel in novel_list for sentence in sent_tokenize(novel)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words_by_sentence = [fast_tokenize(sentence.lower()) for sentence in sentences]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll double check that we don't have any empty sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words_by_sentence = [sentence for sentence in words_by_sentence if sentence != []]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should now have a `list` of `list`s with sentences and words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words_by_sentence[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "\n",
    "# Word2Vec\n",
    "\n",
    "### Word Embeddings\n",
    "Word2Vec is the most prominent word embedding algorithm. Word embedding generally attempts to identify semantic relationships between words by observing them in context.\n",
    "\n",
    "Imagine that each word in a novel has its meaning determined by the ones that surround it in a limited window. For example, in Moby Dick's first sentence, “me” is paired on either side by “Call” and “Ishmael.” After observing the windows around every word in the novel (or many novels), the computer will notice a pattern in which “me” falls between similar pairs of words to “her,” “him,” or “them.” Of course, the computer had gone through a similar process over the words “Call” and “Ishmael,” for which “me” is reciprocally part of their contexts.  This chaining of signifiers to one another mirrors some of humanists' most sophisticated interpretative frameworks of language.\n",
    "\n",
    "The two main flavors of Word2Vec are CBOW (Continuous Bag of Words) and Skip-Gram, which can be distinguished partly by their input and output during training. Skip-Gram takes a word of interest as its input (e.g. \"me\") and tries to learn how to predict its context words (\"Call\",\"Ishmael\"). CBOW does the opposite, taking the context words (\"Call\",\"Ishmael\") as a single input and tries to predict the word of interest (\"me\").\n",
    "\n",
    "In general, CBOW is is faster and does well with frequent words, while Skip-Gram potentially represents rare words better.\n",
    "\n",
    "### Word2Vec Features\n",
    "<ul>\n",
    "<li>`size`: Number of dimensions for word embedding model</li>\n",
    "<li>`window`: Number of context words to observe in each direction</li>\n",
    "<li>`min_count`: Minimum frequency for words included in model</li>\n",
    "<li>`sg` (Skip-Gram): '0' indicates CBOW model; '1' indicates Skip-Gram</li>\n",
    "<li>`alpha`: Learning rate (initial); prevents model from over-correcting, enables finer tuning</li>\n",
    "<li>`iterations`: Number of passes through dataset</li>\n",
    "<li>`batch_words`: Number of words to sample from data during each pass</li>\n",
    "</ul>\n",
    "\n",
    "Note: cell below uses default value for each argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We've gotten accustomed to training powerful models in Python with one line of code, why stop now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = gensim.models.Word2Vec(words_by_sentence, size=100, window=5, \\\n",
    "                               min_count=5, sg=0, alpha=0.025, iter=5, batch_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "We can return the actual high-dimensional vector by simply indexing the model with the word as the key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model['whale']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`gensim` comes with some handy methods to analyze word relationships. `similarity` will give us a number from 0-1 based on how similar two words are. If this sounds like cosine similarity for words, you'd be right! It just takes the cosine similarity of these high dimensional vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.similarity('sense','sensibility')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find cosine distance between two clusters of word vectors. Each cluster is measured as the mean of its words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.n_similarity(['sense','sensibility'],['whale','harpoon'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find words that don't belong with `doesnt_match`. It finds the mean vector of the words in the `list`, and identifies the furthest away:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.doesnt_match(['pride','prejudice', 'harpoon'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most famous implementation of this vector math is semantics. What happens if we take:\n",
    "\n",
    "$$King - Man + Woman = $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Schmidt looked at words associated with male and female pronouns to investigate gender. Let's try take all the female pronouns and subtracting the male pronouns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(positive=['she','her','hers','herself'], negative=['he','him','his','himself'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the opposite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(positive=['he','him','his','himself'], negative=['she','her','hers','herself'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about together (*genderless* in Schmidt's sense)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.most_similar(positive=['she','her','hers','herself','he','him','his','himself'], topn=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Homework\n",
    "\n",
    "Use the `most_similar` method to find the tokens nearest to 'car' in our model. Do the same for 'motorcar'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What characterizes each word in our corpus? Does this make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Vector addition and subtraction can be thought of in terms of analogy. From the example above: 'man' is to 'king' as 'woman' is to '???'. Use the `most_similar` method to find: 'paris' is to 'france' as 'london' is to '???'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "What has our model learned about nation-states?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Perform the canonic Word2Vec addition again but leave out a term. Try 'king' - 'man', 'woman' - 'man', 'woman' + 'king'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "What do these indicate semantically?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Visualization\n",
    "\n",
    "We can use multi-dimensional scaling to visualize this space just like we did with the documents before. But there are a lot of words here, so let's limit it to 50 words from our female gendered subset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "her_tokens = [token for token,weight in model.most_similar(positive=['she','her','hers','herself'], \\\n",
    "                                                       negative=['he','him','his','himself'], topn=50)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to get the vector from each word, just like above, and add that to a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors = [model[word] for word in her_tokens]    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then calculate pairwise the cosine distance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import pairwise\n",
    "dist_matrix = pairwise.pairwise_distances(vectors, metric='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use `MDS` to reduce the dimensions to two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import MDS\n",
    "mds = MDS(n_components = 2, dissimilarity='precomputed')\n",
    "embeddings = mds.fit_transform(dist_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some fancy `matplotlib` code..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_, ax = plt.subplots(figsize=(10,10))\n",
    "ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)\n",
    "for i in range(len(vectors)):\n",
    "    ax.annotate(her_tokens[i], ((embeddings[i,0], embeddings[i,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "What kinds of semantic relationships exist in the diagram above? Are there any words that seem out of place? How do you think they go there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Saving/Loading Models\n",
    "\n",
    "We can save the model as a `.txt` file with the `save_word2vec_format` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.wv.save_word2vec_format('word2vec.txtalb_Novel150_English.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load up a model, we just ask `gensim`. Here's a model trained on Eighteenth Century Collections Online corpus (~2500 texts) made available by Ryan Heuser: http://ryanheuser.org/word-vectors-1/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecco_model = gensim.models.KeyedVectors.load_word2vec_format('data/word2vec.ECCO-TCP.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecco_model.most_similar(positive=['woman', 'king'], negative=['man'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecco_model.most_similar(positive=['she','her','hers','herself'], negative=['he','him','his','himself'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this differ from our novels model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "\n",
    "# Homework\n",
    "\n",
    "Heuser's blog post explores an analogy in eighteenth-century thought that Riches are to Virtue what Learning is to Genius. How true is this in the ECCO-trained Word2Vec model? Is it true in the one we trained?\n",
    "\n",
    "How might we compare word2vec models more generally?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Alternative features for a classification model\n",
    "\n",
    "This is really cool but what implications does this have for our model of language? Well, word embeddings are simply more precise features of what we've been trying to get at already. That means we can use them in the machine learning models we've been building.\n",
    "\n",
    "Recall our DTM bag of words classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score, train_test_split\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "reviews = [movie_reviews.raw(fileid) for fileid in movie_reviews.fileids()]\n",
    "judgements = [movie_reviews.categories(fileid)[0] for fileid in movie_reviews.fileids()]\n",
    "\n",
    "np.random.seed(0)\n",
    "\n",
    "X, y = shuffle(reviews, judgements, random_state=0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=50)\n",
    "\n",
    "# get tfidf values\n",
    "tfidf = TfidfVectorizer()\n",
    "tfidf.fit(X)\n",
    "X_train_transformed = tfidf.transform(X_train)\n",
    "X_test_transformed = tfidf.transform(X_test)\n",
    "\n",
    "# build and test logit\n",
    "logit_class = LogisticRegression(penalty='l2', C=1000)\n",
    "logit_model = logit_class.fit(X_train_transformed, y_train)\n",
    "logit_model.score(X_test_transformed, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "So how can we use word embeddings as features? Believe it or not, one of the most effective ways is to simply average each dimension of our embedding across all the words for a given document. Recall our w2v model for novels was trained for 100 dimensions. Creating the features for a specific document would entail first extracting the 100 dimensions for each word, then average each dimension across all words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean([model[w] for w in fast_tokenize(X[0]) if w in model], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a set `X` array with 100 features. We can write a function to do this for us for any given string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def w2v_featurize(document, model):\n",
    "    return np.mean([model[w] for w in fast_tokenize(document) if w in model], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then featurize all of our documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_w2v = [w2v_featurize(d, model) for d in X_train]\n",
    "X_test_w2v = [w2v_featurize(d, model) for d in X_test]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fit and score the machine learning modle just as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)\n",
    "logit_model = logit_class.fit(X_train_w2v, y_train)\n",
    "logit_model.score(X_test_w2v, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about Heuser's model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_w2v = [w2v_featurize(d, ecco_model) for d in X_train]\n",
    "X_test_w2v = [w2v_featurize(d, ecco_model) for d in X_test]\n",
    "logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)\n",
    "logit_model = logit_class.fit(X_train_w2v, y_train)\n",
    "logit_model.score(X_test_w2v, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool! But wait, what if we wanted to know *why* the model was making decisions. If we ask for the most postive coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argsort(logit_model.coef_[0])[-10:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the negative:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argsort(logit_model.coef_[0])[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the *indices* for the important features. ***But what are these features now?***\n",
    "\n",
    "---\n",
    "\n",
    "Note that using our novels w2v model was not as accurate as a BoW tfidf method. That should be expected given movie review language is likely ***VERY*** different from our novel corpus. And our novel corpus likely didn't even have entries for a lot of the words used in our movie reviews corpus.\n",
    "\n",
    "For modern English, most people look for Stanford's [GloVe](https://nlp.stanford.edu/projects/glove/) model. This was trained on 6 billion tokens from Wikipedia and Gigaword! Quite a step up from 150 novels. Even the smallest model is a bit large to be working with on our cloud server, but using this model and the code below, you can see it's power:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    ">>> os.system('python -m gensim.scripts.glove2word2vec -i glove.6B.100d.txt -o glove.6B.100d.w2v.txt')\n",
    ">>> glove = gensim.models.KeyedVectors.load_word2vec_format('glove.6B.100d.w2v.txt')\n",
    "\n",
    ">>> X_train_glove = [w2v_featurize(d, glove) for d in X_train]\n",
    ">>> X_test_glove = [w2v_featurize(d, glove) for d in X_test]\n",
    "\n",
    ">>> logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)\n",
    ">>> logit_model = logit_class.fit(X_train_glove, y_train)\n",
    ">>> logit_model.score(X_test_glove, y_test)\n",
    "\n",
    ".8125\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this is not as accurate as our BoW *tfidf* method, there have been several applications and transformations of word embeddings that have proven to be more accurate than a BoW *tfidf* on general modern text corpora. And keep in mind, one of the most interesting parts of this is that it only uses 100 dimensions, i.e., we can get ~81% accuracy by reducing a movie review to only 100 different features (our BoW model had over 39000!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_transformed"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}