part3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fjzcvgz2NTzj"
   },
   "source": [
    "# Machine Reading: Advanced Topics in Word Vectors\n",
    "## Part III. Pre-trained Models and Extended Vector Algorithms (50 mins)\n",
    "\n",
    "This is a 4-part series of Jupyter notebooks on the topic of word embeddings originally created for a workshop during the Digital Humanities 2018 Conference in Mexico City. Each part is comprised of a mix of theoretical explanations and fill-in-the-blanks activities of increasing difficulty.\n",
    "\n",
    "Instructors:\n",
    "- Eun Seo Jo, <a href=\"mailto:eunseo@stanford.edu\">*eunseo@stanford.edu*</a>, Stanford University\n",
    "- Javier de la Rosa, <a href=\"mailto:versae@stanford.edu\">*versae@stanford.edu*</a>, Stanford University\n",
    "- Scott Bailey, <a href=\"mailto:scottbailey@stanford.edu\">*scottbailey@stanford.edu*</a>, Stanford University"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lG6sPBChKzgG"
   },
   "source": [
    "This unit will explore the various flavors of word embeddings specifically tailored to sentences, word meaning, paragraph, or entire documents. We will give an overview of pre-trained embeddings including where they can be found, how to use them, and what they're effective for.\n",
    "\n",
    "- 0:00 - 0:15 [Out-of-vocabulary words and pre-trained embeddings](#1.-Out-of-vocabulary-words-and-pre-trained-embeddings)\n",
    "- 0:15 - 0:25 [Activity] Bias in pre-trained historical word embeddings\n",
    "- 0:25 - 0:40 [Extending Vector Algorithms: Text Classification](#2.-Extending-Vector-Algorithms:-Text-Classification)\n",
    "- 0:40 - 0:50 [Activity] Authorship attribution\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-tcd-ksk4_Hp"
   },
   "source": [
    "### 0. Setting Up "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aBDSstL82uNa"
   },
   "source": [
    "Before we get started, let's go ahead and set up our notebook. We will start by importing a few Python libraries that we will use throughout the workshop.\n",
    "\n",
    "#### What are these libraries?\n",
    "\n",
    "1. NumPy: This is a package for scientific computing in python. For us, NumPy is useful for vector operations. \n",
    "2. NLTK: Easy to use python package for text processing (lemmatization, tokenization, POS-tagging, etc.)\n",
    "3. gensim: Built-in word2vec and other NLP algorithms\n",
    "4. fastText: Super fast word embeddings library\n",
    "\n",
    "We will be working with a few sample texts using NLTK's corpus package and other corpor downloaaded to such effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 465
    },
    "colab_type": "code",
    "id": "1gctO2uj_sJd",
    "outputId": "876d9303-9c1a-4be4-f75f-9746c8e0b1bc"
   },
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "import os\n",
    "import sys\n",
    "!pip install Cython  # needed to compile fasttext\n",
    "if not os.path.isfile('requirements.txt'):\n",
    "    !pip install -r https://raw.githubusercontent.com/sul-cidr/dh2018-word-vector-workshops/master/requirements.txt\n",
    "else:\n",
    "    !pip install -r requirements.txt\n",
    "!python -m nltk.downloader all\n",
    "print(\"All done!\", file=sys.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all went well, we should be able now to import the next packages into our workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FuIdxn38Njrq"
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import pickle\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import nltk\n",
    "import gensim\n",
    "import fasttext\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8dVMpegUqNQW"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uBOp_mLvnB-N"
   },
   "source": [
    "### 1. Out-of-vocabulary words and pre-trained embeddings\n",
    "\n",
    "So far, we've seen the power of word embeddings and how easy they are to obtain from your own corpus. In most cases, however, we do not have access to millions of unlabelled documents in our target domain that would allow for training good embeddings from scratch. Training word embeddings is very resource intensive and it may require relatively large corpora for the geometric relationships to be semantically meaningful. Still, there are some issues with regular word-oriented embeddings. To illustrate this, consider the next code that trains on the text from _Alice in Wonderland_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(nltk.corpus.gutenberg.raw('carroll-alice.txt')[0:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the handy `.words()` method in NLTK to access just the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = list(map(str.lower, nltk.corpus.gutenberg.words('carroll-alice.txt')))\n",
    "words[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's train a very simple `word2vec` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [words]\n",
    "model = gensim.models.Word2Vec(\n",
    "    documents,\n",
    "    size=25,\n",
    "    window=5,\n",
    "    min_count=1,\n",
    "    workers=10\n",
    ")\n",
    "model.train(documents, total_examples=len(documents), epochs=10)\n",
    "model.wv['alice']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regardless of whether this model is able to compute semantic similarities or not, word vectors have been computed. However, if you try to look for words that are not in the vocabulary you'll get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    model.wv['google']\n",
    "except KeyError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is known as the Out-Of-Vocabulary (OOV) issue in Word2Vec and similar approaches.\n",
    "\n",
    "Now, you may think, I could get synonyms of the OOV words using something like WordNet, and then look for those words' embeddings. And while that might work in some cases, in others it is not that simple. Two such cases are new-ish words like `facebook` and `google`, or proper names of places, like `Teotihuacan`.\n",
    "\n",
    "One way to solve this issue is to use a different measure of atomicity in your algorithm. In Word2Vec-like approaches, including GloVe, the word is the minimum unit, and as such, when looking for words that are not in the vocabulary there is certainly no vector information for it. In contrast, a different approach could train for sub-word units, for example 3-grams. While not guaranteeing that all words will be covered, a good amount of them might be, due to the fact that it's more likely for all possible trigrams to be included in a large enough corpus than all possible words. This is the approach taken by Facebook's fastText."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import FastText\n",
    "\n",
    "fasttext_model = FastText(documents, size=25, min_count=1)\n",
    "fasttext_model.wv['alice']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext_model.wv['google']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fastText also distributes word vectors pre-trained on [Common Crawl](http://commoncrawl.org/) and [Wikipedia](https://www.wikipedia.org/) for more than 157 languages. These models were trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. They come in binary and text format: binary includes a model ready to use while the text format only contains the actual vectors associated to each word on the training set.\n",
    "\n",
    "Gensim is soon to include a special method to load in these fasText embeddings (not working as of 3.4.0). Just take into account that only the `.bin` format allows for OOV word vectors. For the regular and usually lighter `.vec` format you still would need to load in the vectors, save a binary Gensim model, and load it back in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a couple of examples of using `.vec` from the Somali and the Simplified English Wikipedia corpora available for fastText. These files are loaded in using the regular Gensim `KeyedVectors` word2vec model (`.load_word2vec_format()`), and vectors for out of vocabulary cannot be computed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Somali, the word `xiddigta` (meaning *the star*) should have its own vector avalilable since the word is present in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'wiki.so.vec'\n",
    "if not os.path.isfile(filename):\n",
    "    !echo \"Downloading $filename\"\n",
    "    !curl --progress-bar -Lo $filename https://s3-us-west-1.amazonaws.com/fasttext-vectors/$filename\n",
    "\n",
    "somali_model = gensim.models.KeyedVectors.load_word2vec_format(filename, binary=False)\n",
    "somali_model.wv['xiddigta'][:25]  # it means 'the star' in Somali"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the word `ciyaalsuuq` (meaning *unruly youth*) raises a `KeyError` in the word vectors dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    somali_model.wv['ciyaalsuuq'][:25]\n",
    "except KeyError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same thing occurs in English: while words like `star` are certainly available, words such as `bibliopole` (meaning *a person who buys and sells books, especially rare ones*) are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This might take a while\n",
    "filename = 'wiki.simple.zip'\n",
    "if (not os.path.isfile(filename)\n",
    "        and not os.path.isfile(filename.replace('.zip', '.vec'))\n",
    "        and not os.path.isfile(filename.replace('.zip', '.bin'))):\n",
    "    !echo \"Downloading $filename\"\n",
    "    !curl --progress-bar -Lo $filename https://s3-us-west-1.amazonaws.com/fasttext-vectors/$filename\n",
    "if (os.path.isfile(filename)\n",
    "        and (not os.path.isfile(filename.replace('.zip', '.vec'))\n",
    "                 or not os.path.isfile(filename.replace('.zip', '.bin')))):\n",
    "    !unzip $filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_model = gensim.models.KeyedVectors.load_word2vec_format(\n",
    "    filename.replace('.zip', '.vec'), binary=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_model.wv['star'][:25] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    english_model.wv['bibliopole'][:25] \n",
    "except KeyError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fastText English embeddings **without** sub-word information are also included in Gensim's `downloader` feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader as pretrained\n",
    "\n",
    "pretrained.info()['models']['fasttext-wiki-news-subwords-300']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext_english = pretrained.load('fasttext-wiki-news-subwords-300')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext_english.wv['star'][:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By contrast, when using the `.bin` file and loading it in Gensim using the special `Fastext.load_fasttext_format()` method, out of vocabulary words suddenly have embeddings available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_oov = FastText.load_fasttext_format('wiki.simple')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_oov.wv['bibliopole'][:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "<strong>Activity</strong>\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Could you find a word in the `english_oov` model for which there is no embedding? And in the `english_model` model? Would the embedding for `ciyaalsuuq` be available in any of these models?\n",
    "<br>\n",
    "<em>\n",
    "<!--\n",
    "<strong>Hint</strong>: Use the numpy functions\n",
    "-->\n",
    "</em>\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we've seen, non-existing words in English, such as the Somali `ciyaalsuuq`, also become available, so it's a feature we must be very careful when using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_oov.wv['ciyaalsuuq'][:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsurprisingly, if we check what other words are similar in English to the Somali word `ciyaalsuuq` we get a bunch of words that are not really from English. To be completely fair, the Simple English corpus might not be as reliable as the full English one for finding semantic similarities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_model.similar_by_vector(english_oov.wv['ciyaalsuuq'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### fastText package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While Gensim provides a way to create fastText embeddings with sub-word information and even load fastText pre-trained word embeddings, there is also a standalone tool, `fasttext`, and an accompanying Python library to do the same. Unfortunately, the Python bindings haven't been updated and it seems to be broken when trying to load in binary models generated with newer versions of the fastText command line tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fasttext\n",
    "\n",
    "try:\n",
    "    fasttext.load_model(\"wiki.simple.bin\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other functionalities, such as building embedding from your own corpus using either Skip-gram or CBOW, are available, as well as methods to create text classifiers very easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext.skipgram(nltk.corpus.gutenberg.abspath('carroll-alice.txt'), 'alice_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext.cbow(nltk.corpus.gutenberg.abspath('carroll-alice.txt'), 'alice_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "__label__pos This is some wonderful positive text.\n",
    "__label__neg This is some awful negative text.\n",
    "\"\"\"\n",
    "with open('sentiment_train.txt', 'w') as f:\n",
    "    f.write(text.strip())\n",
    "test = \"\"\"\n",
    "__label__pos This is wonderful.\n",
    "__label__neg This is awful.\n",
    "\"\"\"\n",
    "with open('sentiment_test.txt', 'w') as f:\n",
    "    f.write(test.strip())\n",
    "\n",
    "classifier = fasttext.supervised('sentiment_train.txt', 'sentiment_model')\n",
    "result = classifier.test('sentiment_test.txt')\n",
    "print('Precision:', result.precision)\n",
    "print('Recall:', result.recall)\n",
    "print('Number of examples:', result.nexamples)\n",
    "print('\"This is wonderfully awful\":',\n",
    "      classifier.predict_proba(['This is wonderfully awful'])[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pre-trained vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list of pre-trained word vectors grows every day, and while it's impractical to enumerate them all, some of them are listed below.\n",
    "\n",
    "- English\n",
    "  - fastText. Embeddings (300 dimensions) by Facebook [with](https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki-news-300d-1M-subword.vec.zip) and [without sub-word information](https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki-news-300d-1M.vec.zip) trained on Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (16B tokens), and on [Common Crawl (600B tokens)](https://s3-us-west-1.amazonaws.com/fasttext-vectors/crawl-300d-2M.vec.zip).\n",
    "  - [Google News](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/). Embeddings (300 dimensions) by Google trained on Google News (100B) using word2vec with negative sampling and context window BoW with size ~5 ([link](http://code.google.com/p/word2vec/)). There also fastText versions from 2016 with and without sub-word information for Wikipedia and with no sub-word information for Common Crawl.\n",
    "  - [LexVec](https://github.com/alexandres/lexvec). Embeddings (300 dimensions) trained using LexVec with and without sub-word information trained on Common Crawl, and on Wikipedia 2015 + NewsCrawl.\n",
    "  - Freebase [IDs](https://docs.google.com/file/d/0B7XkCwpI5KDYaDBDQm1tZGNDRHc/edit?usp=sharing) and [names](https://docs.google.com/file/d/0B7XkCwpI5KDYeFdmcVltWkhtbmM/edit?usp=sharing). Embeddings (1000 dimensions) by Google trained on Gooogle News (100B) using word2vec, skip-gram and context window BoW with size ~10 ([link](http://code.google.com/p/word2vec/)).\n",
    "  - [Wikipedia 2014 + Gigaword 5](http://nlp.stanford.edu/data/glove.6B.zip). Embeddings (50, 100, 200, and 300 dimensions) by GloVe trained on Wikipedia data from 2014 and newswire data from the mid 1990s through 2011 using GloVe with AdaGrad and context window 10+10 ([link](http://nlp.stanford.edu/projects/glove/)).\n",
    "  - Common Crawl [42B](http://nlp.stanford.edu/data/glove.42B.300d.zip) and [840B](http://nlp.stanford.edu/data/glove.840B.300d.zip). Embeddings (300 dimensions) by GloVe trained on Common Crawl (42B and 840B) using GloVe and AdaGrad ([link](http://nlp.stanford.edu/projects/glove/)).\n",
    "  - [Twitter (2B Tweets)](http://www-nlp.stanford.edu/data/glove.twitter.27B.zip). Embeddings (25, 50, 100, and 200 dimensions) by GloVe trained on Twitter (27B) using GloVe with GloVe and AdaGrad ([link](http://nlp.stanford.edu/projects/glove/)).\n",
    "  - [Wikipedia dependency](http://u.cs.biu.ac.il/~yogo/data/syntemb/deps.words.bz2). Embeddings (300 dimensions) by Levy & Goldberg trained on Wikipedia 2015 using word2vec modified with word2vec and context window syntactic dependencies ([link](https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/)).\n",
    "  - [DBPedia vectors (wiki2vec)](https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent). Embeddings (1000 dimensions) by Idio trained on Wikipedia 2013 using word2vec with word2vec, skip-gram and context window BoW, 10 ([link](https://github.com/idio/wiki2vec#prebuilt-models)).\n",
    "  - [60 Wikipedia embeddings with 4 kinds of context](http://vsmlib.readthedocs.io/en/latest/tutorial/getting_vectors.html#). Embeddings (25, 50, 100, 250, and 500 dimensions) by Li, Liu et al. trained on Wikipedia using Skip-Gram, CBOW, GloVe with original and modified and context window 2 ([link](http://vsmlib.readthedocs.io/en/latest/tutorial/getting_vectors.html#)).\n",
    "- Multi-lingual\n",
    "  - [fastText](https://fasttext.cc/docs/en/crawl-vectors.html). Embeddigns for 157 languages trained using fastText on Wikipedia 2016 and Common Crawl using CBOW with position-weights, 300 dimensions, with character n-grams of length 5, a window of size 5 and 10 negatives. Both vectors and binary models for OOV are available. There is an old version of these embeddings trained only on Wikipedia 2016 for almost [300 languages](https://fasttext.cc/docs/en/pretrained-vectors.html).\n",
    "  - [BPEemb](https://github.com/bheinzerling/bpemb). Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) on Wikipedia 2017 with sub-word information.\n",
    "  - [Kyubyong's wordvectors](https://github.com/Kyubyong/wordvectors#pre-trained-models). Embeddings with and without sub-word information trained on Wikipedia dumps from 2017 for +30 languages.\n",
    "  - [Polyglot](https://sites.google.com/site/rmyeid/projects/polyglot#h.p_ID_98). Embeddings for more than 100 languages trained on their Wikipedias from 2013. Provides competitive performance with near state-of-art methods in English, Danish and Swedish.\n",
    "\n",
    "There is even a tool, [`chakin`](https://github.com/chakki-works/chakin#supported-vectors), that allows to easily download word vectors with and without sub-word information for 11 languages.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import chakin\n",
    "\n",
    "chakin.search(lang='Japanese')"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "chakin.download(number=6, save_dir='./')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Historical Word Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Humanities, despite the value of word embeddings, we usually want to train our own models or to have access to models that are related to a specific time period of study. It might not be of much help to analyze 19th Century literature with word vectors trained on a Google News corpus, specially since the semantic of the words themselves have been proven to change over time.\n",
    "\n",
    "There is, however, a collection of [historical word vectors](https://nlp.stanford.edu/projects/histwords/) made avaliable to use by the Stanford NLP Group and others (special thanks to [Ryan Heuser](http://ryanheuser.org/)). The embeddings (300 dimensions) are generated using word2vec skip-gram with negative sampling and trained on Google N-Grams for English, English Fiction, French, German, and Simplified Chinese; on the Corpus of Historical American English (COHA); and on the Century Collections Online (ECCO):\n",
    "- English:\n",
    "  - [All English](http://snap.stanford.edu/historical_embeddings/eng-all_sgns.zip) (1800s-1990s by decade)\n",
    "  - [English Fiction](http://snap.stanford.edu/historical_embeddings/eng-fiction-all_sgns.zip) (1800s-1990s by decade)\n",
    "  - [Genre-Balanced American English](http://snap.stanford.edu/historical_embeddings/coha-word_sgns.zip) (1830s-2000s by decade) (COHA)\n",
    "  - [Genre-Balanced American English, word lemmas](http://snap.stanford.edu/historical_embeddings/coha-lemma_sgns.zip) (1830s-2000s) (COHA)\n",
    "  - [ECCO](http://ryanheuser.org/data/word2vec.ECCO.skipgram_n=10.model.txt.gz). Eighteenth Century Collections Online (ECCO), “Literature and Language,” 1700-99, with 1.9 billion words and trained using word2vec with skip-gram size of 10 words\n",
    "  - [ECCO20](https://archive.org/details/word-vectors-18c-word2vec-models-across-20-year-periods). ECCO split in twenty-year periods of 18C, with 150 million words each and trained using word2vec with skip-gram size of 10 words\n",
    "  - [ECCO-TCP](http://ryanheuser.org/data/word2vec.ECCO-TCP.txt.zip). ECCO with 80 million words trained using skip-gram size of 5 words. Also available for [size of 10 words](http://ryanheuser.org/data/word2vec.ECCO-TCP.skipgram_n=10.txt.zip).\n",
    "- Multi-lingual:\n",
    "  - [French](http://snap.stanford.edu/historical_embeddings/fre-all_sgns.zip) (1800s-1990s by decade)\n",
    "  - [German](http://snap.stanford.edu/historical_embeddings/ger-all_sgns.zip) (1800s-1990s by decade)\n",
    "  - [Simplified Chinese](Simplified Chinese (1950-1990s) (1950-1990s by decade)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's download and prepare some of these pre-trained word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Downloading and preparing the pre-trained embeddings\n",
    "pretrained.load('word2vec-google-news-300',\n",
    "                return_path=True)  # return_path avoids to load the model in memory\n",
    "\n",
    "for filename, dirname in (('eng-fiction-all_sgns.zip', 'fiction'),\n",
    "                          ('coha-word_sgns.zip', 'coha')):\n",
    "    if (not os.path.isfile(filename)\n",
    "            and not os.path.isdir(dirname)):\n",
    "        print(f'Downloading {filename}')\n",
    "        !curl --progress-bar -Lo $filename http://snap.stanford.edu/historical_embeddings/$filename\n",
    "    if (os.path.isfile(filename)\n",
    "            and not os.path.isdir(dirname)):\n",
    "        print(f'Uncompressing {filename}')\n",
    "        !unzip -q -o $filename -d $dirname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for corpus, years in (('fiction', (1900, 1950)),  # range(1800, 1991, 10)\n",
    "                      ('coha', (1900, 1950))):  # range(1810, 2001, 10)\n",
    "    for year in tqdm(list(years), desc=f'Generating vector files - {corpus}'):\n",
    "        if os.path.isfile(f'{corpus}/{year}.vec'):\n",
    "            continue\n",
    "        with open(f'{corpus}/{year}.vec', 'w') as vector_file:\n",
    "            vectors = np.load(open(f'{corpus}/sgns/{year}-w.npy', 'rb'))\n",
    "            words = pickle.load(open(f'{corpus}/sgns/{year}-vocab.pkl', 'rb'))\n",
    "            vector_file.write(\"{} {}\".format(*vectors.shape))\n",
    "            for index, word in enumerate(words):\n",
    "                vector = np.array2string(vectors[index],\n",
    "                                         formatter={'float_kind':'{0:.9f}'.format})[1:-1]\n",
    "                vector = vector.replace('\\n', '')\n",
    "                vector_file.write(f'\\n{word} {vector}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have historical word embeddings for English Fiction and COHA for 1900 and 1950, available as `.vec` files as `<fiction|coha>/<year>.vec`. For example, `fiction/1900.vec`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "<strong>Activity</strong>\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Word embeddigns allow for analogy checking. For example, `man is to king as woman is to queen`, expressed as `man:king :: woman:queen`, has its reflection on the vector representions of the words `man`, `king`, `woman`, `queen` in such a way that $\\vec{king} - \\vec{man} + \\vec{woman} \\approx \\vec{queen}$. However, this can also highlight some biases in the specific corpora the model has been trained on. Using as a base the pair `she-he`, find the most similar term for female from the term in the next list: doctor, captain, gallant, sheriff, engineer, scientist, author, surgeon, honorable, philosopher, warrior, architect, magician, liar, and coward.\n",
    "<br>\n",
    "When possible, compute the similarity between the expected term for female and the one for male. Use the Google News (2015), English Fiction (1900, 1950) and Genre-Balanced American English (1900, 1950) embeddings. For example, using the English Fiction 1850 embeddings, `he:gallant :: she:x` solves to `x=gentle`, and the similarity between gallant and gentle is `~0.418`. Can you spot the problem?\n",
    "<br>\n",
    "<em>\n",
    "<strong>Hint</strong>: Use Gensim's `most_similar_cosmul()`/`most_similar()` and `similarity()` functions\n",
    "</em>\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solution (redacted):\n",
    "```python\n",
    "words = ...\n",
    "\n",
    "def get_models():\n",
    "    \"\"\"Yields a tuple of 2 elements: (model name, model instance)\"\"\"\n",
    "    yield ('Fiction 1900', ...)\n",
    "    yield ('Fiction 1950', ...)\n",
    "    yield ('COHA 1900', ...)\n",
    "    yield ('COHA 1950', ...)\n",
    "    yield ('Google News', ...)\n",
    "\n",
    "for (model_name, model) in get_models():\n",
    "    print(model_name)\n",
    "    print(len(model_name) * '=')\n",
    "    for ...:\n",
    "        expected = ...\n",
    "        similarity = ...\n",
    "        print(f\"he:{word:20} :: she:{expected:20}\\tsimilarity = {similarity:.4}\")\n",
    "    print()\n",
    "del model\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output (excerpt):\n",
    "```\n",
    "Fiction 1900\n",
    "============\n",
    "he:doctor               :: she:girl                \tsimilarity = 0.1677\n",
    "he:captain              :: she:major               \tsimilarity = 0.4955\n",
    "he:gallant              :: she:kindest             \tsimilarity = 0.2594\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"right\"><small class=\"text-muted\">*Double click here to see the full code and output solution*</small></div>\n",
    "\n",
    "<!--\n",
    "%%capture --no-stdout\n",
    "words = \"doctor, captain, gallant, sheriff, engineer, scientist, author, surgeon, honorable, philosopher, warrior, architect, magician, liar, coward\"\n",
    "words = words.split(', ')\n",
    "\n",
    "def get_models():\n",
    "    \"\"\"Yields a tuple of 2 elements: (model name, model instance)\"\"\"\n",
    "    yield ('Fiction 1900',\n",
    "        gensim.models.KeyedVectors.load_word2vec_format('fiction/1900.vec', binary=False))\n",
    "    yield ('Fiction 1950',\n",
    "        gensim.models.KeyedVectors.load_word2vec_format('fiction/1950.vec', binary=False))\n",
    "    yield ('COHA 1900',\n",
    "        gensim.models.KeyedVectors.load_word2vec_format('coha/1900.vec', binary=False))\n",
    "    yield ('COHA 1950',\n",
    "        gensim.models.KeyedVectors.load_word2vec_format('coha/1950.vec', binary=False))\n",
    "    yield ('Google News', pretrained.load('word2vec-google-news-300'))\n",
    "\n",
    "for (model_name, model) in get_models():\n",
    "    print(model_name)\n",
    "    print(len(model_name) * '=')\n",
    "    for word in words:\n",
    "        expected = model.most_similar(\n",
    "            positive=['she', word], negative=['he'])[0][0]\n",
    "        similarity = model.similarity(expected, word)\n",
    "        print(f\"he:{word:20} :: she:{expected:20}\\tsimilarity = {similarity:.4}\")\n",
    "    print()\n",
    "del model\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Extending Vector Algorithms: Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Averaging vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen that vectors for out of vocabulary words are obtained by splitting the word into its n-grams, getting the embedding for the n-grams, and then averaging the composition to produce the final word vector for the OOV word.\n",
    "\n",
    "\n",
    "What is best in the word2vec approach is that operations on the vectors approximately keep the characteristics of the words, so that joining (averaging) vectors from the words of a sentence produce a vector that is likely to represent the general topic of the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, the same technique used for OOV words in fastText can also be used to produce embeddings for sentences, paragraphs and even entire documents, making it possible for text classification purposes.\n",
    "\n",
    "> The goal of text classification is to assign documents (such as emails, posts, text messages, product reviews, etc...) to one or multiple categories. Such categories can be review scores, spam v.s. non-spam, or the language in which the document was typed. Nowadays, the dominant approach to build such classifiers is machine learning, that is learning classification rules from examples. In order to build such classifiers, we need labeled data, which consists of documents and their corresponding categories (or tags, or labels).\n",
    "-- [fastText documentation](https://fasttext.cc/docs/en/supervised-tutorial.html#what-is-text-classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this way of seeing sentences and documents might actually work. Consider the next dummy sentiment texts with 1 positive sentence and 1 negative sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_sentence = 'This is some wonderful positive text'\n",
    "negative_sentence = 'This is some awful negative text'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the embeddings for every single word then average them per sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [positive_sentence.split(), negative_sentence.split()]\n",
    "model = gensim.models.Word2Vec(\n",
    "    documents,\n",
    "    size=25,\n",
    "    window=5,\n",
    "    min_count=1,\n",
    "    workers=10\n",
    ")\n",
    "model.train(documents, total_examples=len(documents), epochs=10)\n",
    "model.wv['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_vectors = [model.wv[word]\n",
    "                    for word in positive_sentence.split()]\n",
    "negative_vectors = [model.wv[word]\n",
    "                    for word in negative_sentence.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_vectors[:2]  # first 2 words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's get the average vectors por positive and negative sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_vector = np.mean(positive_vectors, axis=0)\n",
    "negative_vector = np.mean(negative_vectors, axis=0)\n",
    "positive_vector, negative_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now run the same process for a couple of test sentences and see if their sentence vectors are similar to the positive or the negative one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "for test_sentence in ('This is awful', 'This is wonderful'):\n",
    "    test_vector = np.mean([model.wv[word] for word in test_sentence.split()], axis=0)\n",
    "    print(test_sentence)\n",
    "    print('\\tSimilarity to positive sentence',\n",
    "          cosine_similarity(positive_vector.reshape(1, -1), test_vector.reshape(1, -1))[0][0])\n",
    "    print('\\tSimilarity to negative sentence',\n",
    "          cosine_similarity(negative_vector.reshape(1, -1), test_vector.reshape(1, -1))[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although this approach seems naive, it's still part of the way fastText does its text classification. Moreover, it uses a shallow neural network and ideas similar to CBOW but for word n-grams. The result, which rivals state-of-the-art text classification techniques based on deep learning, runs several orders of magtinude faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Gensim, this functionality is under `gensim.models.Doc2Vec`, and it uses a slightly different approach based on Word2Vec, the *Paragraph Vector*, where the model learns to correlate labels and words, rather than words with other words.\n",
    "\n",
    "> The idea is straightforward: we act as if a paragraph (or document) is just another vector like a word vector, but we will call it a paragraph vector. We determine the embedding of the paragraph in vector space in the same way as words. Our paragraph vector model considers local word order like bag of n-grams, but gives us a denser representation in vector space compared to a sparse, high-dimensional representation. -- [RaRe Technologies](http://nbviewer.jupyter.org/github/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-IMDB.ipynb#Paragraph-Vector)\n",
    "\n",
    "The first step is coming up with a vector that represents the *meaning* of a document, which can then be used as input to a supervised machine learning algorithm to associate documents with labels. There are 2 flavors that are roughly the equivalents of CBOW and Skip-gram. In the **Paragraph Vector Distributed Memory (PV-DM)** model, analogous to CBOW Word2vec, the paragraph vectors are obtained by training a neural network on the fake task of inferring a center word based on context words and a context paragraph. A paragraph is a context for all words in the paragraph, and a word in a paragraph can have that paragraph as a context. In the **Paragraph Vector Distributed Bag of Words (PV-DBOW)** model, analogous to Skip-gram Word2vec, the paragraph vectors are obtained by training a neural network on the fake task of predicting a probability distribution of words in a paragraph given a randomly-sampled word from the paragraph.\n",
    "\n",
    "Doc2Vec, which considers blocks of texts and units, has built-in support for the equivalent for CBOW as *distributed memory* (`dm`), and *distributed bag of words* (`dbow`) for Skip-gram. Since the distributed memory model performed noticeably better in the paper, that algorithm is the default when running Doc2Vec. You can still force the `dbow` model if you wish, by using the `dm=0` flag in constructor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see an example where we aim to build a classifier for Jane Austen and G.K. Chesterton's works, where each work is assigned a label or tag with its author. Let's also suppose we don't know who the author for Austen's *Emma* was but we know for sure that is either Austen or Chesterton. One way to approach this is by making building a classifier to predict a label or tag for the unseen anonymous work so we can see who the classifier thinks the work belongs to. This is, with some licenses, a very basic instance of authorship attribution.\n",
    "\n",
    "We start by obtaining the total number of words and sentences for all their works in the NLTK's Gutenberg corpus and consider that to be all the works that they ever wrote."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "works = ['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt',\n",
    "         'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt']\n",
    "for work in works:\n",
    "    print(\"{:25} {:8} words {:5} sentences\".format(\n",
    "        work,\n",
    "        len(nltk.corpus.gutenberg.words(work)),\n",
    "        len(nltk.corpus.gutenberg.sents(work)))\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By removing *Emma* from our list of works, we roughly have the same amount of words per author, which should alleviate the class imbalance issue.\n",
    "\n",
    "| Austen's Work          | Words    | Sentences  |\n",
    "| ---------------------- |:--------:| ----------:|\n",
    "| austen-persuasion.txt  |  98171   |  3747      |\n",
    "| austen-sense.txt       |  141576  |  4999      |\n",
    "| **Total**              |**239747**|**8746**    |    \n",
    "\n",
    "| chesterton's Work      | Words    | Sentences  |\n",
    "| ---------------------- |:--------:| ----------:|\n",
    "| chesterton-ball.txt    |  96996   |  4779      |\n",
    "| chesterton-brown.txt   |  86063   |  3806      |\n",
    "| chesterton-thursday.txt|  69213   |  3742      |\n",
    "| **Total**              |**252272**|**12327**   |    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim provides a couple of classes to encode sentences and entire documents: `LabeledSentence()` and `TaggedDocument()`, respectively. While their operation is very similar, the former is intended for short texts and sentences, while the latter performs better for large chunks of text.\n",
    "\n",
    "Once we have the words from our corpus of works, let's create instances of each of them as `TaggedDocument`s by passing the list of words and the label or tag. We will use the label `anonymous` for Austen's *Emma*. Similarly, we could have built `documents` to have a list of sentences instead and then predict the proportion of sentences in Austen's *Emma* properly assigned to her."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.doc2vec import TaggedDocument\n",
    "    \n",
    "documents = [\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('austen-persuasion.txt'),\n",
    "        tags=['austen']),\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('austen-sense.txt'),\n",
    "        tags=['austen']),\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('chesterton-ball.txt'),\n",
    "        tags=['chesterton']),\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('chesterton-brown.txt'),\n",
    "        tags=['chesterton']),\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('chesterton-thursday.txt'),\n",
    "        tags=['chesterton']),\n",
    "    TaggedDocument(\n",
    "        words=nltk.corpus.gutenberg.words('austen-emma.txt'),\n",
    "        tags=['anonymous']),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, it's time to initialize a `Doc2Vec()` model with some learning rate `alpha`, build the vocabulary from our list of works, and start training over a number of epochs, 10 in this case. The values for `total_examples` and `epochs` are the default ones, although for some reason `Dov2Vec()` forces you to pass them in explicitly: `total_examples=model.corpus_count` and `epochs=model.epochs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Doc2Vec\n",
    "\n",
    "model = Doc2Vec(min_cout=0)\n",
    "model.build_vocab(documents)\n",
    "for epoch in tqdm(range(10), desc='Epochs'):\n",
    "    model.train(documents, total_examples=model.corpus_count, epochs=model.epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model has been fit to out corpus, we have available the usual functions from word2vec but for documents, `docs2vec`. Let's see, from our list of tags that represent the authors of the works, which one is more similar to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.docvecs.most_similar_to_given('anonymous', ['austen', 'chesterton'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's compute similarity and distance from Austen's Emma (`anonymous`) to the vectors that represent Jane Austen and Chesterton."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for author in ('austen', 'chesterton'):\n",
    "    print(author)\n",
    "    print(\"\\tSimilarity\", model.docvecs.similarity('anonymous', author))\n",
    "    print(\"\\tDistance\", model.docvecs.distance('anonymous', author))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, there are many interpretations of this and it's, by no means, the best way to tackle the problem of authorship attribution. However, it gives a sense of how extending ideas from word2vec can be leveraged to other problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "<strong>Activity</strong>\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Using the example code given for fastText text clasification, replicate the Austen and Chesterton classifier.\n",
    "You can use a sentence level approach, holding 25% of all sentences for testing and assessing the performance.\n",
    "Once the model has been fit, we'll use it to predict the percentage of sentences from Austen's *Emma* that fall to either Austen or Chesterton.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solution (redacted):\n",
    "```python\n",
    "# 1. Build the list of sentences, prefixing each one with __label__author\n",
    "texts = []\n",
    "for work in works:\n",
    "    for sent in ...:\n",
    "        if 'emma' not in work:  # Our anonymous work!\n",
    "            author = ...\n",
    "            sentence = ...\n",
    "            texts.append(f\"__label__{author} {sentence}\")\n",
    "\n",
    "# 2. Randomly split the list in training and testing\n",
    "training = ...\n",
    "testing = ...\n",
    "\n",
    "# 3. Get the texts from the training and testing lists\n",
    "training_text = ...\n",
    "testing_text = ...\n",
    "\n",
    "# 4. Create the temporary files so fastText can read them \n",
    "with open('author_train.txt', 'w') as f:\n",
    "    f.write(training_text)\n",
    "with open('author_test.txt', 'w') as f:\n",
    "    f.write(testing_text)\n",
    "\n",
    "# 5. Build a classifier and test it on the testing sentences\n",
    "classifier = fasttext....\n",
    "result = classifier...\n",
    "print('Precision:', result...)\n",
    "print('Recall:', result...)\n",
    "print('Number of sentences:', result...)\n",
    "\n",
    "# 6. For each sentence in Austen's Emma, predict its author\n",
    "emma_sentences = ...\n",
    "predictions = classifier...\n",
    "\n",
    "# 7. Print the percentage of sentences from Austen' Emma associated to each author\n",
    "authors = ...\n",
    "counts = ...\n",
    "print()\n",
    "print(\"Predicting Austen's Emma's sentences\")\n",
    "for author, count in zip(*[authors, counts]):\n",
    "    print(f\"{author.capitalize()} accounts for {100*count:.2f}% of sentences.\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output:\n",
    "```\n",
    "Precision: 0.8980831277282216\n",
    "Recall: 0.8980831277282216\n",
    "Number of sentences: 5269\n",
    "\n",
    "Predicting Austen's Emma's sentences\n",
    "Austen accounts for 80.38% of sentences.\n",
    "Chesterton accounts for 19.62% of sentences.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"right\"><small class=\"text-muted\">*Double click here to see the full code solution*</small></div>\n",
    "\n",
    "<!--\n",
    "\n",
    "#- 1. Build the list of sentences, prefixing each one with __label__author\n",
    "texts = []\n",
    "for work in works:\n",
    "    for sent in nltk.corpus.gutenberg.sents(work):\n",
    "        if 'emma' not in work:  # Our anonymous work!\n",
    "            author = work.split('-')[0]\n",
    "            sentence = ' '.join(sent)\n",
    "            texts.append(f\"__label__{author} {sentence}\")\n",
    "\n",
    "#- 2. Randomly split the list in training and testing\n",
    "training_size = int(np.floor(len(texts) * 0.75))\n",
    "np.random.shuffle(texts)\n",
    "training, testing = texts[:training_size], texts[training_size:]\n",
    "\n",
    "#- 3. Get the texts from the training and testing lists\n",
    "training_text = '\\n'.join(training).strip()\n",
    "testing_text = '\\n'.join(testing).strip()\n",
    "\n",
    "#- 4. Create the temporary files so fastText can read them \n",
    "with open('author_train.txt', 'w') as f:\n",
    "    f.write(training_text)\n",
    "with open('author_test.txt', 'w') as f:\n",
    "    f.write(testing_text)\n",
    "\n",
    "#- 5. Build a classifier and test it on the testing sentences\n",
    "classifier = fasttext.supervised('author_train.txt', 'author_model')\n",
    "result = classifier.test('author_test.txt')\n",
    "print('Precision:', result.precision)\n",
    "print('Recall:', result.recall)\n",
    "print('Number of sentences:', result.nexamples)\n",
    "\n",
    "#- 6. For each sentence in Austen's Emma, predict its author\n",
    "emma_sentences = [' '.join(sent)\n",
    "                  for sent in nltk.corpus.gutenberg.sents('austen-emma.txt')]\n",
    "predictions = classifier.predict(emma_sentences)\n",
    "\n",
    "#- 7. Print the percentage of sentences from Austen' Emma associated to each author\n",
    "authors, counts = np.unique(predictions, return_counts=True)\n",
    "counts = counts / sum(counts)\n",
    "print()\n",
    "print(\"Predicting Austen's Emma's sentences\")\n",
    "for author, percentage in zip(*[authors, counts]):\n",
    "    print(f\"{author.capitalize()} accounts for {100*percentage:.2f}% of sentences.\")\n",
    "-->"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "dh2018_workshop_wordvectors.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}