labs/06_deep_nlp/Character_Level_Language_Model_rendered.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Character-level Language Modeling with LSTMs\n",
    "\n",
    "This notebook is adapted from [Keras' lstm_text_generation.py](https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py).\n",
    "\n",
    "Steps:\n",
    "\n",
    "- Download a small text corpus and preprocess it.\n",
    "- Extract a character vocabulary and use it to vectorize the text.\n",
    "- Train an LSTM-based character level language model.\n",
    "- Use the trained model to sample random text with varying entropy levels.\n",
    "- Implement a beam-search deterministic decoder.\n",
    "\n",
    "\n",
    "**Note**: fitting language models is compute intensive. It is recommended to do this notebook on a server with a GPU or powerful CPUs that you can leave running for several hours at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Loading some text data\n",
    "\n",
    "Let's use some publicly available philosophy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpus length: 600893 characters\n"
     ]
    }
   ],
   "source": [
    "from keras.utils.data_utils import get_file\n",
    "\n",
    "URL = \"https://s3.amazonaws.com/text-datasets/nietzsche.txt\"\n",
    "\n",
    "corpus_path = get_file('nietzsche.txt', origin=URL)\n",
    "text = open(corpus_path).read().lower()\n",
    "print('Corpus length: %d characters' % len(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preface\n",
      "\n",
      "\n",
      "supposing that truth is a woman--what then? is there not ground\n",
      "for suspecting that all philosophers, in so far as they have been\n",
      "dogmatists, have failed to understand women--that the terrible\n",
      "seriousness and clumsy importunity with which they have usually paid\n",
      "their addresses to truth, have been unskilled and unseemly methods for\n",
      "winning a woman? certainly she has never allowed herself to be won; and\n",
      "at present every kind of dogma stands with sad and discouraged mien--if,\n",
      "indeed, it stands at all! for there are scoffers who maintain that it\n",
      "has fallen, that all dogma lies on the gro ...\n"
     ]
    }
   ],
   "source": [
    "print(text[:600], \"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text = text.replace(\"\\n\", \" \")\n",
    "split = int(0.9 * len(text))\n",
    "train_text = text[:split]\n",
    "test_text = text[split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Building a vocabulary of all possible symbols \n",
    "\n",
    "To simplify things, we build a vocabulary by extracting the list all possible characters from the full datasets (train and validation).\n",
    "\n",
    "In a more realistic setting we would need to take into account that the test data can hold symbols never seen in the training set. This issue is limited when we work at the character level though.\n",
    "\n",
    "Let's build the list of all possible characters and sort it to assign a unique integer to each possible symbol in the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total chars: 56\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "`char_indices` is a mapping to from characters to integer identifiers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(char_indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(' ', 0),\n",
       " ('!', 1),\n",
       " ('\"', 2),\n",
       " (\"'\", 3),\n",
       " ('(', 4),\n",
       " (')', 5),\n",
       " (',', 6),\n",
       " ('-', 7),\n",
       " ('.', 8),\n",
       " ('0', 9),\n",
       " ('1', 10),\n",
       " ('2', 11),\n",
       " ('3', 12),\n",
       " ('4', 13),\n",
       " ('5', 14)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(char_indices.items())[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "`indices_char` holds the reverse mapping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(indices_char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ä'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indices_char[52]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "While not strictly required to build a language model, it's a good idea to have a look at the distribution of relative frequencies of each symbol in the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0sAAADFCAYAAACFFmlDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHT9JREFUeJzt3X20JVV55/HvL90giPLeIUpDmkSWhuBSoUUSYiYBxRaM\nMDOYYFSIIRKXENHoTNqZyeAQyWqTrDg6MSZEegBDRIIoREAkgOIbLw0oryIdQGkGpQXEKFFEn/mj\ndsdjW33Py72Hvt39/ax11q3aZ++zn7rnrZ7aVfukqpAkSZIk/bif2tgBSJIkSdJ8ZLIkSZIkST1M\nliRJkiSph8mSJEmSJPUwWZIkSZKkHiZLkiRJktTDZEmSJEmSepgsSZIkSVIPkyVJkiRJ6rFwYwcw\n13bddddasmTJxg5DkiRJ0jx1/fXXf6OqFg2rt9klS0uWLGHVqlUbOwxJkiRJ81SSr4xSz9PwJEmS\nJKmHyZIkSZIk9TBZkiRJkqQeJkuSJEmS1GNospRkZZIHktwyULZzksuS3Nn+7tTKk+Q9SVYnuSnJ\nfgNtjm3170xy7ED5/klubm3ekyQz9SFJkiRJT4RRZsM7A/gr4KyBsuXA5VW1Isnytv5HwEuBvdvt\nBcD7gBck2Rk4GVgKFHB9kgur6uFW53XANcDFwDLgkhn62GQsWX7RyHXvWXH4FCORJEmSNK6hI0tV\ndRXw0HrFRwBntuUzgSMHys+qztXAjkmeBrwEuKyqHmoJ0mXAsnbf9lV1dVUVXUJ25JA+JEmSJGnq\nJr1mabequr8tfw3YrS3vDtw7UG9NK5upfE1P+Ux9/IQkxydZlWTV2rVrJ9gcSZIkSfpxs57goY0I\n1RzEMnEfVXVaVS2tqqWLFg39IV5JkiRJGmrSZOnr7RQ62t8HWvl9wB4D9Ra3spnKF/eUz9SHJEmS\nJE3dpMnShcC6Ge2OBS4YKD+mzYp3IPBIO5XuUuDQJDu1We0OBS5t930ryYFtFrxj1nusvj4kSZIk\naeqGzoaX5IPArwG7JllDN6vdCuDcJMcBXwF+s1W/GDgMWA08CrwWoKoeSvInwHWt3ilVtW7SiDfQ\nzbi3Ld0seJe08g31IUmSJElTNzRZqqpXbuCuQ3rqFnDCBh5nJbCyp3wVsG9P+YN9fUiSJEnSE2HW\nEzxIkiRJ0ubIZEmSJEmSepgsSZIkSVIPkyVJkiRJ6mGyJEmSJEk9TJYkSZIkqYfJkiRJkiT1MFmS\nJEmSpB4mS5IkSZLUw2RJkiRJknqYLEmSJElSD5MlSZIkSephsiRJkiRJPUyWJEmSJKmHyZIkSZIk\n9TBZkiRJkqQeJkuSJEmS1MNkSZIkSZJ6mCxJkiRJUg+TJUmSJEnqYbIkSZIkST1MliRJkiSpx6yS\npSRvTnJrkluSfDDJNkn2SnJNktVJPpRk61b3SW19dbt/ycDjvK2V35HkJQPly1rZ6iTLZxOrJEmS\nJI1j4mQpye7AG4GlVbUvsAA4Gngn8K6qegbwMHBca3Ic8HArf1erR5J9WrtfBJYBf51kQZIFwHuB\nlwL7AK9sdSVJkiRp6mZ7Gt5CYNskC4EnA/cDBwPntfvPBI5sy0e0ddr9hyRJKz+nqr5XVXcDq4ED\n2m11Vd1VVY8B57S6kiRJkjR1EydLVXUf8BfAV+mSpEeA64FvVtXjrdoaYPe2vDtwb2v7eKu/y2D5\nem02VP4TkhyfZFWSVWvXrp10kyRJkiTp383mNLyd6EZ69gKeDmxHdxrdE66qTquqpVW1dNGiRRsj\nBEmSJEmbmdmchvci4O6qWltV3wfOBw4Cdmyn5QEsBu5ry/cBewC0+3cAHhwsX6/NhsolSZIkaepm\nkyx9FTgwyZPbtUeHALcBVwJHtTrHAhe05QvbOu3+K6qqWvnRbba8vYC9gWuB64C92+x6W9NNAnHh\nLOKVJEmSpJEtHF6lX1Vdk+Q84AbgceBG4DTgIuCcJO9oZae3JqcDH0iyGniILvmhqm5Nci5dovU4\ncEJV/QAgyYnApXQz7a2sqlsnjVeSJEmSxjFxsgRQVScDJ69XfBfdTHbr1/0u8IoNPM6pwKk95RcD\nF88mRkmSJEmaxGynDpckSZKkzZLJkiRJkiT1MFmSJEmSpB4mS5IkSZLUw2RJkiRJknqYLEmSJElS\nD5MlSZIkSephsiRJkiRJPUyWJEmSJKmHyZIkSZIk9TBZkiRJkqQeJkuSJEmS1MNkSZIkSZJ6mCxJ\nkiRJUg+TJUmSJEnqYbIkSZIkST1MliRJkiSph8mSJEmSJPUwWZIkSZKkHiZLkiRJktTDZEmSJEmS\nepgsSZIkSVKPWSVLSXZMcl6SLyW5PckvJdk5yWVJ7mx/d2p1k+Q9SVYnuSnJfgOPc2yrf2eSYwfK\n909yc2vzniSZTbySJEmSNKrZjiy9G/h4VT0LeA5wO7AcuLyq9gYub+sALwX2brfjgfcBJNkZOBl4\nAXAAcPK6BKvVed1Au2WzjFeSJEmSRjJxspRkB+BXgdMBquqxqvomcARwZqt2JnBkWz4COKs6VwM7\nJnka8BLgsqp6qKoeBi4DlrX7tq+qq6uqgLMGHkuSJEmSpmrhLNruBawF/m+S5wDXAycBu1XV/a3O\n14Dd2vLuwL0D7de0spnK1/SU/4Qkx9ONVrHnnntOvkXzwJLlF41U754Vh085EkmSJGnLNpvT8BYC\n+wHvq6rnAd/hR6fcAdBGhGoWfYykqk6rqqVVtXTRokXT7k6SJEnSFmA2ydIaYE1VXdPWz6NLnr7e\nTqGj/X2g3X8fsMdA+8WtbKbyxT3lkiRJkjR1EydLVfU14N4kz2xFhwC3ARcC62a0Oxa4oC1fCBzT\nZsU7EHikna53KXBokp3axA6HApe2+76V5MA2C94xA48lSZIkSVM1m2uWAP4AODvJ1sBdwGvpErBz\nkxwHfAX4zVb3YuAwYDXwaKtLVT2U5E+A61q9U6rqobb8BuAMYFvgknaTJEmSpKmbVbJUVV8Alvbc\ndUhP3QJO2MDjrARW9pSvAvadTYySJEmSNInZ/s6SJEmSJG2WTJYkSZIkqYfJkiRJkiT1MFmSJEmS\npB4mS5IkSZLUw2RJkiRJknqYLEmSJElSD5MlSZIkSephsiRJkiRJPUyWJEmSJKmHyZIkSZIk9TBZ\nkiRJkqQeJkuSJEmS1MNkSZIkSZJ6mCxJkiRJUg+TJUmSJEnqYbIkSZIkST1MliRJkiSpx8KNHYBm\nb8nyi0aqd8+Kw6cciSRJkrT5cGRJkiRJknqYLEmSJElSj1knS0kWJLkxycfa+l5JrkmyOsmHkmzd\nyp/U1le3+5cMPMbbWvkdSV4yUL6sla1Osny2sUqSJEnSqOZiZOkk4PaB9XcC76qqZwAPA8e18uOA\nh1v5u1o9kuwDHA38IrAM+OuWgC0A3gu8FNgHeGWrK0mSJElTN6tkKcli4HDg/W09wMHAea3KmcCR\nbfmItk67/5BW/wjgnKr6XlXdDawGDmi31VV1V1U9BpzT6kqSJEnS1M12ZOl/A/8V+GFb3wX4ZlU9\n3tbXALu35d2BewHa/Y+0+v9evl6bDZX/hCTHJ1mVZNXatWtnuUmSJEmSNItkKcnLgAeq6vo5jGci\nVXVaVS2tqqWLFi3a2OFIkiRJ2gzM5neWDgJenuQwYBtge+DdwI5JFrbRo8XAfa3+fcAewJokC4Ed\ngAcHytcZbLOhckmSJEmaqolHlqrqbVW1uKqW0E3QcEVVvQq4EjiqVTsWuKAtX9jWafdfUVXVyo9u\ns+XtBewNXAtcB+zdZtfbuvVx4aTxSpIkSdI4ZjOytCF/BJyT5B3AjcDprfx04ANJVgMP0SU/VNWt\nSc4FbgMeB06oqh8AJDkRuBRYAKysqlunEO8Wacnyi0aqd8+Kw6cciSRJkjQ/zUmyVFWfBD7Zlu+i\nm8lu/TrfBV6xgfanAqf2lF8MXDwXMWp2TK4kSZK0pZnGyJIEmGBJkiRp02aypHnFBEuSJEnzxWx/\nZ0mSJEmSNkuOLGmTNupIFDgaJUmSpPE4siRJkiRJPUyWJEmSJKmHyZIkSZIk9TBZkiRJkqQeJkuS\nJEmS1MNkSZIkSZJ6mCxJkiRJUg+TJUmSJEnqYbIkSZIkST1MliRJkiSpx8KNHYD0RFuy/KKR6t2z\n4vBZtZEkSdKmzZElSZIkSephsiRJkiRJPUyWJEmSJKmHyZIkSZIk9TBZkiRJkqQeJkuSJEmS1MNk\nSZIkSZJ6TJwsJdkjyZVJbktya5KTWvnOSS5Lcmf7u1MrT5L3JFmd5KYk+w081rGt/p1Jjh0o3z/J\nza3Ne5JkNhsrSZIkSaOazcjS48Bbqmof4EDghCT7AMuBy6tqb+Dytg7wUmDvdjseeB90yRVwMvAC\n4ADg5HUJVqvzuoF2y2YRryRJkiSNbOJkqarur6ob2vK/ArcDuwNHAGe2amcCR7blI4CzqnM1sGOS\npwEvAS6rqoeq6mHgMmBZu2/7qrq6qgo4a+CxJEmSJGmqFs7FgyRZAjwPuAbYrarub3d9DditLe8O\n3DvQbE0rm6l8TU95X//H041Wseeee06+IdIcWrL8opHq3bPi8ClHIkmSpEnMeoKHJE8BPgy8qaq+\nNXhfGxGq2fYxTFWdVlVLq2rpokWLpt2dJEmSpC3ArJKlJFvRJUpnV9X5rfjr7RQ62t8HWvl9wB4D\nzRe3spnKF/eUS5IkSdLUzWY2vACnA7dX1V8O3HUhsG5Gu2OBCwbKj2mz4h0IPNJO17sUODTJTm1i\nh0OBS9t930pyYOvrmIHHkiRJkqSpms01SwcBrwFuTvKFVvbfgBXAuUmOA74C/Ga772LgMGA18Cjw\nWoCqeijJnwDXtXqnVNVDbfkNwBnAtsAl7SZJkiRJUzdxslRVnwE29LtHh/TUL+CEDTzWSmBlT/kq\nYN9JY5Q2JaNOCAFOCiFJkvREmPUED5IkSZK0OTJZkiRJkqQeJkuSJEmS1MNkSZIkSZJ6zGY2PEkb\n2aiTQgxOCDFJG0mSpC2RI0uSJEmS1MORJUlDORolSZK2RCZLkuacyZUkSdoceBqeJEmSJPUwWZIk\nSZKkHp6GJ2le8NQ9SZI035gsSdpkmWBJkqRpMlmStMUYNbmCHyVY/paVJElbLq9ZkiRJkqQeJkuS\nJEmS1MNkSZIkSZJ6eM2SJM0DXuckSdL8Y7IkSZugSSarkCRJ4/E0PEmSJEnq4ciSJG0hPNVPkqTx\nOLIkSZIkST0cWZIkbZCjUZKkLdm8T5aSLAPeDSwA3l9VKzZySJKkGYybYJmQSZLmq3mdLCVZALwX\neDGwBrguyYVVddvGjUyStDGZYEmSngjzOlkCDgBWV9VdAEnOAY4ATJYkSWOZ1ojXJG0Gk7gnos2W\nuC3zNa71+5E0v6WqNnYMG5TkKGBZVf1eW38N8IKqOnG9escDx7fVZwJ3PKGBbtiuwDc2kzbG5bZs\nDtsyX+OapM18jWuSNsbltmwO2zJf45qkzXyNa5I2xjU/t2U++NmqWjS0VlXN2xtwFN11SuvWXwP8\n1caOa4z4V20ubYzLbZlvfWxOcbktxuW2zK8+Nqe43Bbjmo/bsind5vvU4fcBewysL25lkiRJkjRV\n8z1Zug7YO8leSbYGjgYu3MgxSZIkSdoCzOsJHqrq8SQnApfSTR2+sqpu3chhjeO0zaiNcU2/zXyN\na5I2xjX9NvM1rknaGNf028zXuCZpY1zTbzNf45qkjXFNv80kfWwy5vUED5IkSZK0scz30/AkSZIk\naaMwWZIkSZI2Q0kOSvKrGzuOTZnJ0iYsyY5J3jBBu89NI57NUZIlSW6ZoN23Z9Hn25O8ddL2cy3J\nG5PcnuTsjR3L5mbS19d8tbltz6Sm/Rk7zmdEkmcl+VySm5N8KsmuI7bbOslVSca6tjnJtq2fBUPq\nrUzywObwekmyR5Irk9yW5NYkJ02hj22SXJvki62P/zVG2wVJbkzysRHr39NeL19IsmrENm9ucd2S\n5INJthlSf8ck5yX5Uvt++aVR+pm2JM9s273u9q0kbxrS5qS23bcOqzvQZlmSO5KsTrJ8xDavT3JM\nkuVJXj5im+cBrwU+P0r9SSR5UZJTkrx81G3Z1Jgsbdp2BMZOlqrql6cQyyYnHd8Dw70BeHFVvWpj\nB6LNT9sxW5Lkkxs7lrkyDz9jX11VzwY+B7x+lAZV9RhwOfBbY/b1u8D5VfWDIfXOAJaN+dgTS3LP\nFB/+ceAtVbUPcCBwQpJ95riP7wEHV9VzgOcCy5IcOGLbk4Dbx+zv16vquVW1dFjFJLsDbwSWVtW+\ndBNyHT2k2buBj1fVs4DnjBJf+5z4tyRfGB7+v7fZtiU9j41yoKCq7mjb/Vxgf+BR4CMzPP6+wOuA\nA9p2vCzJM4bEtAB4L/BSYB/glaO8Xqrqb6rqrKpaUVUjzQxdVTdW1e9V1fdHqT+JqvrnqvqfVXVh\nVa2YVj8bkzuK80SSV7ejRl9I8rfDjso1K4Cfb23+fIy+xhr1SPLRJNe3oybHj1D/D9tRlltGOcrS\nPgBvT/J3rY9PJNl2SJvtklzUjrLdkmSkL/TW1x1JzgJu4cd/x2tDFowT2ySS/PckX07yGeCZI7Y5\nJslN7X/wgSF1f+yIf5K3Jnn7CH38DfBzwCVJ3jzk8b+U5Iy2HWe3o02fTXJnkgNGaHt2ex2cl+TJ\nQ+L64/Y8fqYdxZzxKHs76vWmgfVTM8PR3yT/Jckb2/K7klzRlg/O3I+wLRxz21ckOWFgfV6NRK6T\n5OfSHc1+/hw+5vPba36b9hlwa9tZmZokFyd5+phtRvqMHXzfj/I6nkRVfamq7mqrTwK+O0bzjwLj\nHiR5FXDBCHFdBTw05mM/IdIdwV83snB3kitnql9V91fVDW35X+l2/Hefy5iqs+51tVW7DZ2hK8li\n4HDg/XMZT4+FwLbpRiKfDPy/GWLaAfhV4HToEvOq+uaI/fxLS2RGUlX/1upvMJ4ZHNL6+8oMdX4B\nuKaqHq2qx4FPAf9pyOMeAKyuqrvaQYlzgCOGBZPkn8bcFxt5v3JgP++2JG9J8o6B/Yvfn6t+Nlkb\n+1dxvRV0b7Z/ArZq638NHDNCuyXALRP09+0x6+/c/m5Ll2DsMkPd/YGbge2ApwC3As8bYTseB57b\n1s+lOxI6U5v/DPzdwPoOI27LEuCHwIFj1B8rtnH/xwP/sycD2wOrgbcOafOLwJeBXQefo1FfK8Bb\ngbePGN896/oZ4f/0bLqDMNcDK4HQfQl8dEjbAg5q6ytn2n7g+cAXgG2ApwJ3jvD/WgLc0JZ/CviX\nIa/jA4F/bMufBq6l2zk5Gfj9UZ/bEV9fI297q/M84FMD67cBe8xVTHOwPbfQJfw3As8Zoc11dAct\nzh+xj3cAf0F3ZPZtY8R2MfD0J+j/MPT9P8n7fqDt20etO9DmJXQ78TuO0WYBsHaM+lsDXxv39TJi\n3U+39/36txeN2P66CZ7HrVq/vzHmNn0V2H4Kr6sFbZu/DbxzxDbntdfarwEfG7HN3cANdJ/jx4/Y\n5qQW11rg7CF1n9s+U89onxPvB7aby9dLT9t7GPI91tNmJXDikDq/QPddvEt7L38e+D9D2hwFvH9g\n/TXAX40Qzzj7YmPtVw489lbtM/nUtr51W99rLvrZVG+OLM0Ph9B9mF2Xbnj5ELqj+fPFG5N8Ebia\nbqdm7xnq/grwkar6TnVHwc4HXjhCH3dX1bqh9evpPhRncjPw4iTvTPLCqnpkhD7W+UpVXT1G/XFj\nG9cL6f5nj1bVtxjth5cPptuZ/wZAVc2HI7R3V9XNVfVDuiT58uo+PW9m+P/s3qr6bFv+e7rX0YYc\nBFxQVd+t7ijuPw0LrKruAR5Md/72ocCNVfXgDE2uB/ZPsj3d6S+fB5bSPVefHtbfmMbZdqrqRuCn\nkzw9yXOAh6vq3jmOaTYW0Y0svKqqvjisclU9v6ruraphR2PXOQV4Md3z8WejBlVVh1XVJEeXp2WS\n9/1E0p1ufDrw8hr9CD7VnUr3WJKnjthkV2Dkxx9HVb2w2ulR693+ecT2k4xwvhu4oqqGfsYAJHkK\n8GHgTe05nVNV9YPqRkkWAwcMG1VN8jLggaq6fsyufqWq9qM7TeyEDJkcIMlOdAfF9gKeDmyX5NUz\nNFkI7Ae8r6qeB3wHmFfXuiTZGng58I8z1auq24F3Ap8APk6XzA47BXVS4+yLjbtf+YZW73rg54Hf\nauvX0r2vN9TXfN9/nRPz+kdptyABzqyqt23sQNaX5NeAFwG/VFWPpruuYMYLNyf0vYHlH9AdOdmg\nqvpykv2Aw4B3JLm8qk4Zsa/vTDO2eepxfvy022k/hz8cWP8hwz9r1j+dZBo/APd+4HeAn6E7Yrjh\nYKq+n+TuVv9zwE3ArwPPYPxz/4eZZNv/ke7o5M8AHxqlk3bq3uva6jQTh0fojqz/Ct2o11zbhW7U\neiu61/G47+dNXlW9fcwmTwceqao7J+hunFP3/o3pfLaQ5NN0I8nre+uoCdOY/f0O8LPAiSPW34ou\nUTq7qs6f63gGVdU326mBy+hGGDbkIODlSQ6je162T/L3VTVTIkNV3df+PpDkI3SnjV01Q5MX0R0s\nWwuQ5Hzgl+kO/vRZA6ypqmva+nnMs2SJLlG8oaq+PqxiVZ1OO6UwyZ/Sbd9M7uPHLwFY3Mo2aIJ9\nsZH3K5P8B7rX0i+3x/4w8A9V9eFhbcfpZ1PmyNL8cDlwVJKfBkiyc5KfHaHdv9L/5TGXdqA7cv1o\nkmfRnZ40k08DRyZ5cpLtgP/I3B+Jp11D8GhV/T3w53RHqTZVV9H9z7ZtR3B/Y4Q2VwCvSLILdK+Z\nIfW/TjcasUuSJwEvm1XEc2/P/Gg2pN8GPjND3c8Cv9GuW3kKo2/LR+i+EJ4PXDpC/U/Tna54VVt+\nPd2I1CjXCVye7qLnUYyz7et8iO4C6qMYcuRznap678DR+GmOsDxG974/JslvT+Hx/xb4Y+BsuiO6\nUzXmczmOSd73k3oYeMu4jdrnyzdqxIvDq+phums85zxhmu3I0jiS7E/33n91GykfVj90O8u3V9Vf\nznU8rY9FSXZsy9vSja5+aaY2VfW2qlpcVUvoPi+uGJYotWsBn7pumW4kftiMhV8FDmzf+6EbXdjg\nQaWq+hpwb5J11+cewnQOrMzGK4EPjlJxYN9tT7rrlf5hSJPrgL2T7NVGsI5m+MjyuPti4+xX7kh3\nMOXRVmc/4PfbvsK6GQK3m4N+NlkmS/NAVd0G/A/gE0luAi4DnjZCuweBz6ab4GDkCR7G9HG6C9Bv\np5tQYsbT16q7yPUMuqHba+jOy71xCnE9G7i2DfueTHcdwyap/c8+BHwRuITug3RYm1uBU4FPtWH5\nGb+g287OKXTPy2UM+ZLdCO6gO93jdmAn4H0bqlhV19F9sdxE9/+6mW40Y0bVXUh7JXBuDZ+pC7oE\n6WnA59vRxe8yQuLfTnl6BqNfvD7ytq/Tnv+nAvdV1f0j9jO2TDC5AUBVfYcuiX1zRpzidsR4jgG+\nX1X/QPd59PwkB4/YdpKJGsZ9LtcZmlBP8r4fiOv17X8xqh2A3xuj/jq/Dlw0ZptPMORUUoAkH6Q7\nvfWZSdYkOW6C+KblRGBn4Mp20fqwyREOorvu5OD8aGKIw+Y4pqe1eG6ie61cVlUjTQU+pt2Az7Tv\nlWuBi6rq4zM1aCNE59Fd53Qz3b7laUP6+QPg7LY9zwX+dLaBz5WWGLyY7jKCUXw4yW10p4SfMOxU\n1+omgjiR7qDd7XTfSbcO6WPcfbFx9isvAX7QHvsDdAny54Cb0k0M9Tds4OyQSfdfNzUZ4SCpJE1N\nkiV0Fx6PPKtZkqdU1bfTzRx3Fd1FyDcMafNTdF/mr5jwdKRRY9sX+N2q+sNp9aEnxiTPZRuNuaGq\nxjq6mm52ym9X1V+MF+X0tNOpllfVl8dosx/w5qp6zfQi05Zoku+Kgbb30E1t/o05DktbAK9ZkrQp\nOi3d71JsQ3e+9LBEaR/gY3QX1E8tUQKoqlsAE6XNwLjPZRu5+iTdbH2btHZ60EfHSZSgGzFL9wOt\nC0Ycwd0itaT68p67Dhky+YykJ5gjS5IkSZrXkuxBd3rYgzXiby2167s+TzdL57Nrfswcq02MyZIk\nSZIk9XCCB0mSJEnqYbIkSZIkST1MliRJkiSph8mSJEmSJPX4/0oLSAQGKi+aAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f62fc6ca860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "counter = Counter(text)\n",
    "chars, counts = zip(*counter.most_common())\n",
    "indices = np.arange(len(counts))\n",
    "\n",
    "plt.figure(figsize=(14, 3))\n",
    "plt.bar(indices, counts, 0.8)\n",
    "plt.xticks(indices, chars);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's cut the dataset into fake sentences at random with some overlap. Instead of cutting at random we could use a English specific sentence tokenizer. This is explained at the end of this notebook. In the mean time random substring will be good enough to train a first language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nb train sequences: 180255\n",
      "nb test sequences: 6005\n"
     ]
    }
   ],
   "source": [
    "max_length = 40\n",
    "step = 3\n",
    "\n",
    "\n",
    "def make_sequences(text, max_length=max_length, step=step):\n",
    "    sequences = []\n",
    "    next_chars = []\n",
    "    for i in range(0, len(text) - max_length, step):\n",
    "        sequences.append(text[i: i + max_length])\n",
    "        next_chars.append(text[i + max_length])\n",
    "    return sequences, next_chars    \n",
    "\n",
    "\n",
    "sequences, next_chars = make_sequences(train_text)\n",
    "sequences_test, next_chars_test = make_sequences(test_text, step=10)\n",
    "\n",
    "print('nb train sequences:', len(sequences))\n",
    "print('nb test sequences:', len(sequences_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's shuffle the sequences to break some of the dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "sequences, next_chars = shuffle(sequences, next_chars,\n",
    "                                random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'distrust, a refinement of distrust of ev'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'e'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next_chars[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Converting the training data to one-hot vectors\n",
    "\n",
    "Unfortunately the LSTM implementation in Keras does not (yet?) accept integer indices to slice columns from an input embedding by it-self. Let's use one-hot encoding. This is slightly less space and time efficient than integer coding but should be good enough when using a small character level vocabulary.\n",
    "\n",
    "**Exercise:** \n",
    "\n",
    "One hot encoded the training `data sequences` as `X` and `next_chars` as `y`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "n_sequences = len(sequences)\n",
    "n_sequences_test = len(sequences_test)\n",
    "voc_size = len(chars)\n",
    "\n",
    "X = np.zeros((n_sequences, max_length, voc_size),\n",
    "             dtype=np.float32)\n",
    "y = np.zeros((n_sequences, voc_size), dtype=np.float32)\n",
    "\n",
    "X_test = np.zeros((n_sequences_test, max_length, voc_size),\n",
    "                  dtype=np.float32)\n",
    "y_test = np.zeros((n_sequences_test, voc_size), dtype=np.float32)\n",
    "\n",
    "# TODO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/language_model_one_hot_data.py\n",
    "n_sequences = len(sequences)\n",
    "n_sequences_test = len(sequences_test)\n",
    "voc_size = len(chars)\n",
    "\n",
    "X = np.zeros((n_sequences, max_length, voc_size),\n",
    "             dtype=np.float32)\n",
    "y = np.zeros((n_sequences, voc_size), dtype=np.float32)\n",
    "\n",
    "X_test = np.zeros((n_sequences_test, max_length, voc_size),\n",
    "                  dtype=np.float32)\n",
    "y_test = np.zeros((n_sequences_test, voc_size), dtype=np.float32)\n",
    "\n",
    "\n",
    "for i, sequence in enumerate(sequences):\n",
    "    for t, char in enumerate(sequence):\n",
    "        X[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1\n",
    "    \n",
    "for i, sequence in enumerate(sequences_test):\n",
    "    for t, char in enumerate(sequence):\n",
    "        X_test[i, t, char_indices[char]] = 1\n",
    "    y_test[i, char_indices[next_chars_test[i]]] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(180255, 40, 56)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(180255, 56)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       ..., \n",
       "       [ 1.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Measuring per-character perplexity\n",
    "\n",
    "The NLP community measures the quality of probabilistic model using [perplexity](https://en.wikipedia.org/wiki/Perplexity).\n",
    "\n",
    "In practice perplexity is just a base 2 exponentiation of the average negative log2 likelihoods:\n",
    "\n",
    "$$perplexity_\\theta = 2^{-\\frac{1}{n} \\sum_{i=1}^{n} log_2 (p_\\theta(x_i))}$$\n",
    "\n",
    "**Note**: here we define the **per-character perplexity** (because our model naturally makes per-character predictions). **It is more common to report per-word perplexity**. Note that this is not as easy to compute the per-world perplexity as we would need to tokenize the strings into a sequence of words and discard whitespace and punctuation character predictions. In practice the whitespace character is the most frequent character by far making our naive per-character perplexity lower than it should be if we ignored those.\n",
    "\n",
    "**Exercise**: implement a Python function that computes the per-character perplexity with model predicted probabilities `y_pred` and `y_true` for the encoded ground truth:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def perplexity(y_true, y_pred):\n",
    "    \"\"\"Compute the per-character perplexity of model predictions.\n",
    "    \n",
    "    y_true is one-hot encoded ground truth.\n",
    "    y_pred is predicted likelihoods for each class.\n",
    "    \n",
    "    2 ** -mean(log2(p))\n",
    "    \"\"\"\n",
    "    # TODO\n",
    "    return 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/language_model_perplexity.py\n",
    "def perplexity(y_true, y_pred):\n",
    "    \"\"\"Compute the perplexity of model predictions.\n",
    "    \n",
    "    y_true is one-hot encoded ground truth.\n",
    "    y_pred is predicted likelihoods for each class.\n",
    "    \n",
    "    2 ** -mean(log2(p))\n",
    "    \"\"\"\n",
    "    likelihoods = np.sum(y_pred * y_true, axis=1)\n",
    "    return 2 ** -np.mean(np.log2(likelihoods))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2565790685485896"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_true = np.array([\n",
    "    [0, 1, 0],\n",
    "    [0, 0, 1],\n",
    "    [0, 0, 1],\n",
    "])\n",
    "\n",
    "y_pred = np.array([\n",
    "    [0.1, 0.9, 0.0],\n",
    "    [0.1, 0.1, 0.8],\n",
    "    [0.1, 0.2, 0.7],\n",
    "])\n",
    "\n",
    "perplexity(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A perfect model has a minimal perplexity of 1.0 (negative log likelihood of 0.0):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity(y_true, y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Building recurrent model\n",
    "\n",
    "Let's build a first model and train it on a very small subset of the data to check that it works as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Dense\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(128, input_shape=(max_length, voc_size)))\n",
    "model.add(Dense(voc_size, activation='softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(optimizer=optimizer, loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's measure the perplexity of the randomly initialized model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def model_perplexity(model, X, y, verbose=0):\n",
    "    predictions = model.predict(X, verbose=verbose)\n",
    "    return perplexity(y, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55.851569769252848"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_perplexity(model, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's train the model for one epoch on a very small subset of the training set to check that it's well defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4056 samples, validate on 451 samples\n",
      "Epoch 1/1\n",
      "4056/4056 [==============================] - 1s - loss: 3.1151 - val_loss: 3.0674\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f62cc5260b8>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_train = slice(0, None, 40)\n",
    "\n",
    "model.fit(X[small_train], y[small_train], validation_split=0.1,\n",
    "          batch_size=128, nb_epoch=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.06967878041857"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_perplexity(model, X[small_train], y[small_train])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18.217444707632556"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_perplexity(model, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Sampling random text from the model\n",
    "\n",
    "Recursively generate one character at a time by sampling from the distribution parameterized by the model:\n",
    "\n",
    "$$\n",
    "p_{\\theta}(c_n | c_{n-1}, c_{n-2}, \\ldots, c_0) \\cdot p_{\\theta}(c_{n-1} | c_{n-2}, \\ldots, c_0) \\cdot \\ldots \\cdot p_{\\theta}(c_{0})\n",
    "$$\n",
    "\n",
    "The temperature parameter makes it possible to remove additional entropy (bias) into the parameterized multinoulli distribution of the output of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sample_one(preds, temperature=1.0):\n",
    "    \"\"\"Sample the next character according to the network output.\n",
    "    \n",
    "    Use a lower temperature to force the model to output more\n",
    "    confident predictions: more peaky distribution.\n",
    "    \"\"\"\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    # Draw a single sample (size=1) from a multinoulli distribution\n",
    "    # parameterized by the output of the softmax layer of our\n",
    "    # network. A multinoulli distribution is a multinomial\n",
    "    # distribution with a single trial with n_classes outcomes.\n",
    "    probs = np.random.multinomial(1, preds, size=1)\n",
    "    return np.argmax(probs)\n",
    "\n",
    "\n",
    "def generate_text(model, seed_string, length=300, temperature=1.0):\n",
    "    \"\"\"Recursively sample a sequence of chars, one char at a time.\n",
    "    \n",
    "    Each prediction is concatenated to the past string of predicted\n",
    "    chars so as to condition the next prediction.\n",
    "\n",
    "    Feed seed string as a sequence of characters to condition the\n",
    "    first predictions recursively. If seed_string is lower than\n",
    "    max_length, pad the input with zeros at the beginning of the\n",
    "    conditioning string.\n",
    "    \"\"\"\n",
    "    generated = seed_string\n",
    "    prefix = seed_string\n",
    "\n",
    "    for i in range(length):\n",
    "        # Vectorize prefix string to feed as input to the model:\n",
    "        x = np.zeros((1, max_length, voc_size))\n",
    "        shift = max_length - len(prefix)\n",
    "        for t, char in enumerate(prefix):\n",
    "            x[0, t + shift, char_indices[char]] = 1.\n",
    "\n",
    "        preds = model.predict(x, verbose=0)[0]\n",
    "        next_index = sample_one(preds, temperature)\n",
    "        next_char = indices_char[next_index]\n",
    "\n",
    "        generated += next_char\n",
    "        prefix = prefix[1:] + next_char\n",
    "    return generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'philosophers are tee eeee e e ee tee te ee see ee eae e e te e e e esee eee e eeeee e teeeae te eeeee e tee ee ee e es  e ee ee ee  e te eeee ee e e  e eeaeeee ee e e e etee tee e eeeeeeees e ee e iteee tee e ee e e tee teee teeese e teeeeee ee e  eetee e e ee e e tee ite e e tee eeteie eeeeeee eeeese eeeeteeetee te'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_text(model, 'philosophers are ', temperature=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'atheism is the root of f infsnttreetfincesi aaemi eahrtaseiiop sta   tsfeeihsaisnssmeerhtsrxhpcwserohdssrthif2itelulhpayeorrertso insfy aahluiisrie t echfoyelefneanmy nns siterefeesiecsfrlo lti,le l eso shnsotnetd,eeahsc ftne oplvhoedd s  naa  e ycciashe-dsiothye penonstaceworpoto tetsertah so ts  o,ccmcietanelscof ae otn'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_text(model, 'atheism is the root of ', temperature=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training the model\n",
    "\n",
    "Let's train the model and monitor the perplexity after each epoch and sample some text to qualitatively evaluate the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Epoch 1/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.9803 - val_loss: 1.6954\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 5.403\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the same and the surious of the surious of the still a philosophy of the surious of the still the soul the strunce of the soul the strunce of the surious of the surious and the strunkes the same and the soul the soul, a philosophy of the same the soul the soul the still as the soul the soul of the s\n",
      "\n",
      "atheism is the root of the soul of the surious and the strunken and the soul the soul the strund and the soul the still a part the soul, the surious of the same the soul, the strunk of the supter and the strunkes of the soul of the soul of the suphing the soul of the soul of the surious and the soul the strunce of the sam\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are and disiment very in the concerstions of the the there as the cause the cantartion of as the stirnt, in the extenting that the sain ame destinct and have the gersance, he wordgity, is the whilose beto a with morality, and a promonce in in the ristact of the sactions of the same that every of sable a\n",
      "\n",
      "atheism is the root of the saure of the extruntly have and the his the soul of the surious all a ther as the there as the part ham in the oden and the soul, as the soul of the suristind, a be geness of the there a philosophy the there the there and a ward of the roming of the his of the confect and lomen of the there and \n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are in sore wat there be  thrusess ar instanne. bens he all lot notes on inserteniamed--what bace the abtle and industingenssion bastire, all poour ot eideminasomy, ploped a matist, and heal cull peroplisi-smiting homest mights paise it of dactasy amy of the courcasticinm which the rradues and grated re\n",
      "\n",
      "atheism is the root of the simaus, prepairess, beabting, and ruplings but a nrich houtted with a surforible oun harmsule all wornd a desimarily hus to consturinily sulffhoronemed waid with is it, on un them our our be the dain meanatisiund hod impor that all moral yalrest and nasause to virtuest, the maded youw sen. as, t\n",
      "\n",
      "# Epoch 2/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.6265 - val_loss: 1.5973\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.939\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the states and the same the states and the states and and a superfory and an a superforing and and an a signific and propessing and more the states and suporities of the starn the start and a superfory the stare and a superfory and and the states and a more and the startity and the start and more th\n",
      "\n",
      "atheism is the root of the same the start of the sacrifice the sensible and an a sertous of the starn the starn the starn the man and and an a signous and and an a significal with the superfory and the problems and a superfory and the starn the speak of the sacrifice of the sacrifice and the states and problement, and the\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are must be to the exist of the station with the different and supophition and sentiment--where he poount and morality and and would for the superforwand in with at the life sy that its astants and in the prosent in the were the sppease of the still the power and there is the same and superfoul more sup\n",
      "\n",
      "atheism is the root of not in the conceal the man is no thus also in the econace in the varition man and an instinction of more the are and sufferent and more the good a strong more musto have the everything the artion and philosopher, and everything in the more but at a man for antoocation, in reading all which a distain\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are anliwdelemed be imporfection a miscactical supoenouss\" remorting the anboisy; almost cossedves itsiconing--how deed man in faction--can as a sensions obanistesous elever, evsintifies in the for  more erount belood of the eary and muchist differentsd sconceiring. it i too natureod: almost also, no de\n",
      "\n",
      "atheism is the root of the pargening life, and fulle may as a light reader that themsenve of other as this mappens of will as a moslictily to rong to ancisting most with these lasguaslize--this in by as bad.=--the powerioning sump of may, withistly mustates out of more thre know to, towards!-itave of a sulfsticarions, to \n",
      "\n",
      "# Epoch 3/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.5243 - val_loss: 1.5523\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.784\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the stands that is the standed to the stands to the stands the standed to the standed to the stands and an and the standed to the morality of the stands the stands the stands that is the stands the stands that it is the stands the standed to the stands the stands the stands the stands to the stands \n",
      "\n",
      "atheism is the root of the standed to the same the stands the stands the standed to the stands the stands that is the standed to the sense of the stands the sentiments and an and the stands and an and an an and the stands to the standed to the standed to the stands the stands and the stands the stands the stands that is t\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are in the religions in we suffer tangers of the realing and one and in one and former conception existed of the \"judgments and it is the power of his the the relight to the sentical just in the subourly the understinct of them nould be an and would be to be has all to be the can in general and his exus\n",
      "\n",
      "atheism is the root of the standed, the still for the morality of the is any problem of called that which the tought being in the problem of the most at the same time to leart is the conception of the consequences of the concerning, and a princip and problem that it will not be soul has to the mankind, and who call and al\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are and interacce only the con exponsibile, exploses at\" only their \"excenctive belies appowestars its sud. the he who not in scholes huntites,\" an the \"yot its sowe the inteamen and it is respected unavanty, it ow  was an advence anys prasce seeked to other conceblying to seady--and to soment it will t\n",
      "\n",
      "atheism is the root of the ourselves tanking however, an that perion, un the spirit. the last in 2y peects. the kven dotforting that i how an i more detiveral that in does too; mot powers a lift, thad so recognize forts. the long the cisitact nafully, nature, so libltyet which always to any them like do that these in the \n",
      "\n",
      "# Epoch 4/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.4690 - val_loss: 1.5417\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.685\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the state of the probably the self-soul the stande the stande that is the self-soul all the sense of the sense of the conceal of the conceal the state of the possible to the stande the state of the sense of the stande that is the self-for the stande that is the sense of the sense of the stande to th\n",
      "\n",
      "atheism is the root of the stande that is the stande the stande that is the sense of the stande that is the state of the sense of the stande the state of the stande the standed to the state of the sense of the probably the sense of the stande the stande the sense of the stande to the most the stande that is the sense of t\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are a respectan of the conceal with something to these all the possible conceal in the more will not be respectation of the orgen to sympathy, and procises for truth to the something to the problem of the same conscience of can more the decient will of a soul conceal the word to sacrifice and strange th\n",
      "\n",
      "atheism is the root of \"more concearsely and call about a long to far the most condemned that be present of logen that the scientific of more palloches.  1ules all the stande has all the conceal of moral the spirit for the more the conceal of the present and devilas, and sight as it has been thought in the misunderstond t\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are conception, on thought; it accoudsity for eunly of corvided of its a delugles to didute. all the caue to bilthe,-for hering and bas, thingbles can has in commestived to appunged to the rediounises titure, of havist a questionant really reductding among ity the acopantake and honspined of tacking at \n",
      "\n",
      "atheism is the root of whilosouan again no predicatestical indeed of racse for men, was consequence betray itself painbegecking, at lacked efoegation, aws it cause, man anatable how then to one must be thinker, with so out of being so landnessble of just as for afounbeakes to mansuly insileble of faelly the lose to avolit\n",
      "\n",
      "# Epoch 5/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.4339 - val_loss: 1.5186\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.523\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the state of the same continued and the standand and the state of the state of the same continued and the state of the same time and subjectic and the same continued and such a probably the standand and the state of the state of the same conscience of the standand and the standand and and the subjec\n",
      "\n",
      "atheism is the root of the state of the same conscience of the same conscience and the sense and and and constant and any the state of the same continued and the state of the same conscience that is a probably probably the continued and such a moral for the standand and and the state of the same continued and the state of\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the consident constantly themselves of the soul, the speaks and constant the soufger understand in the contrast and christianing and the consident the moral of the fact that it is the propacity and distrust in the contrary of the higher would be the sublimation of the soul, such a preveality and fee\n",
      "\n",
      "atheism is the root of thee to a providial in which he can even it is not for his continued the continued strength of a probably at the standaques of the south the spent of the might and such a love and adyrochess and surjusifice of artion the parting the possion, who courace such a read for the sentiments of the can and \n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are the ounious procaus which hope as into pontoches, patiened, all and nothing and guist cnomisled his naturalice elsuulates for religtion and ne satterce, a honourisc, we his whole very the considented ussmention that is almost knowledge so create say. that a geciani will, the never hifderreobfude. th\n",
      "\n",
      "atheism is the root of the physiose, possence--num of the christianity to farwisily and trudinused as with the arifglase ensluince of our su ritroa\"e--for ady transticien--the sentious the udees and ampless? herevercal mind become rumld in the land doce actioning timak in all only oper themselves. each for bemin undervran\n",
      "\n",
      "# Epoch 6/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.4075 - val_loss: 1.5150\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.594\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the philosophy of the power and as the possible to the possible of the power and as the possible of the power and as the possible of the possible of the power and as a philosophy and such as the possible of the subtlent to the possible of the possible of the possible of the possible of the possible \n",
      "\n",
      "atheism is the root of the possible of the power and the morals to the morals to the destruction of the power and as the possible of the possible of the possible of the possible of the possible of the power and as the possible of the possible of the possible of the possible of the possible of the possible of the possible \n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are endured by the philosophic to the subtlers its worth as the superstity of the aspectation of the possible as and fore the morals and as the subtlent about all fortion and love the possible of philosophy, the proposition, and of the epinished of are minders to him we seeked you he who has an action o\n",
      "\n",
      "atheism is the root of morality, such a not forter to along in our worth, the stacted the last as a sugported, who has been the europey in the thing to deceived that the higher of interpretted, shame\" and will with a in the infinied in the soul. they owe the respectal effects is it not as a philosophy, or will to man case\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are beroged, and feeld in community of so-utpossion as a revided he cannot and herought the neboly pullious will senfectuous, who who pulsux of outity, se nowisaits that willing charracity that ty ravausedly endolition of theirism a stability. full all history thas the pleasure, all, he possess with are\n",
      "\n",
      "atheism is the root of ministabled so unyreal.\" \"the more woman, or comms the man of litting, that which, when that person, for the chrmoutar more that discainrist, and nound may mochis and the standst, olthese morality is anl owmentsed in retain mightly repaideding that he respect, troew that it can natualishel of beseci\n",
      "\n",
      "# Epoch 7/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3871 - val_loss: 1.5169\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.582\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are and and a present of the stronger of the stronger of the strong and always the sentiments of the sentiments of the sentiments of the sense of the sentiments of the stronger of the subjection of the state of the sentiments of the sentiments of the sentiments of the sentiments of the stronger of the s\n",
      "\n",
      "atheism is the root of the strong and always sense of the sentiments of the same than the sentiments of the sentiments of the strong and always the sentiments of the strong and are a strong and and and as the sense of the sentiments of the sentiments of the sentiments of the stronger of the sentiments of the sentiments of\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are that is have a comprehensible of the condition of the truths of the sentiments of being and sees and cain all the whole fing, in extential that in the consequence of the same former to a strange, and that the disinteresting to the subjections of the strange, that many the thought of the present of t\n",
      "\n",
      "atheism is the root of the free whole and that in the presency of the state of deapong of our a promined and from the outhers of the consideration. the society in the fact of the subject for a man desiration of the deliver to the conditionally who have free complete themselves to the state of the strong of the man, betres\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are naturald sirablameing, originated and  least musicaul othergan were would questions which understanding.\"liage alone of the phomaoced--admadyss adeas, general pirtaables whomere. they even to that an ner or toward tyss a outioned, is a gristyssove, knind of schopenhauer by noumabini in the hubestalg\n",
      "\n",
      "atheism is the root of mankindisone. ald short way a have their whather, even in the rangess and about take that the semman of which and that they knows of many by eximply and man to our body, seemedity of vicies was not ungenimation beguimable \"prevoled to a german knowledge laigre nous thus not beneftering, we pour dist\n",
      "\n",
      "# Epoch 8/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3735 - val_loss: 1.5141\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.599\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the strength of the strength of the strength, and the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the streng\n",
      "\n",
      "atheism is the root of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength of the strength\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are as much that the morality from the more can for the conditional fear of the acullest in the same greatest the from the word it is free shill such a principal and what who is a right in more the scientific the strength of the most the religion with the bad, in the own things to the standaxical profou\n",
      "\n",
      "atheism is the root of the greatfulness is the actions that the have of the sace a necessary and in remarted more his word the strength, and in the actions, or a new can is a virtues, and by the greated by the man deep at the strength of the south the exception, and a possibly rest and not in worker and obsignts. they are\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are thought, but at littlety is it is are that they hover-mannrous the question, from the rever such his steeming in the immediated you; its aids the latter darty in a xiter and that they dild \"to only pro--of doubtful, our histomed, to his concetion of life how powers its oplatitic ri(sners awhy is the\n",
      "\n",
      "atheism is the root of mankind is prig is becauso hives hors inaresuble, even hich use, because by medince from the banking, attew shill revalors and even consite themselves an mean oftments of callot yit do of the presence,  screvirely its the , without an whole right theman, the more and fack the sin an aribshle suachmo\n",
      "\n",
      "# Epoch 9/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3602 - val_loss: 1.4980\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.485\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the sense of the states and an and and a still the sense of the sense of the states and sense of the states and the sense of the states and and the sense of the same things and an and and a strange of the states and sense of the same the states and and an and and in the sense of the states and and a\n",
      "\n",
      "atheism is the root of the states and as the states and and the sense of the sense of the sense of the states and as the sense of the states and the sense of the sense of the sense of the states and as the preserfing and and a still the states and as the states and the same the soul and a strange and and as the sense of t\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are and his nor and has a philosophy, the profound the case of the same ourselved, which we may be the read and prevajing and man as in a still the mastery of the really sense, and develop! the problemmatism that a would as in the rare, and which may is has it needst spiritunism of guisarly on his sense\n",
      "\n",
      "atheism is the root of an and and and man is as other subjection of a good origined by in the gradation of connesssed and predective ourselves as in the most believe the stand and again as the organizm of the spiritun in the motives and see so grated the still the consequence before the man live the conceally the most rea\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are mign is naturality of all a minddequality,\" lightnements of the stand time and are haring \"pride\" of scoll to the mos imporerie sentimed instincts, it. the chimition though let to the higher point--an abso a prowsis of a marration of the good of any the profound, peoharity presinces, who feols, bur \n",
      "\n",
      "atheism is the root of catones the mereal and liver, superfus, scaunnce then other some form onle height,\" the presces in already and nuever, in remained, out of our once the distrenged the music as torxes mature rebuising that whether perhaps a nearn that it to say, most believer for the faction. that the indicence litt(\n",
      "\n",
      "# Epoch 10/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3478 - val_loss: 1.4986\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.560\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the stands the stands to the statemest of the stands to the statemest of the statems of the statemest of the statemest of the statemest of the statemest of the statemest of the statems of the statemest of the statemest of the statemest of the statemest of the statemest of the stands to the stands to\n",
      "\n",
      "atheism is the root of the statems and and of the statemest of the statemest of the statemest of the statemest of the stands to the stands to the statemest of the statemest of the statems and as the stands the statemest of the statemest of the statems and as the stands the statements of the statemest of the statemest of t\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are deception is the responsible to the conception is an one were the artinal should no found how must be faston man has been believe the empronation of philosophy of small as a prose to stands to all the distance, is a philosophy in the same morality and of the into the contradices with they are that t\n",
      "\n",
      "atheism is the root of the probabliar of the taste to a statement, that we like with the masless--the standrance, and now something the same to strength in the man in the stands at life and with, and of the spirit the relation and and standsly conscience, to the and forment, and who say the aspect, as the cause the statem\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are too accustepthe to those. the rutten-man of wands as inreprises personabine again which deepents. ;iloing refess and arrady rather a neadentd, in the mes and happin continue tortheress from in their greatestness, he will tocking--whet betoes, and noncridious accow find its lived belode, valuable how\n",
      "\n",
      "atheism is the root of \"here regard to ours ble, to make all cognition of dend art lull, and who only is aristry, one must wants and disleing \"it! one is by who passed and duncerod,--the abymos, the muces from the concess apparentabaks) evontion, and should earth of himself. under the strongered way as it to sense, affers\n",
      "\n",
      "# Epoch 11/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3365 - val_loss: 1.5042\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.565\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are not the state of the strange of the strength of the strength of the state of the sense and destrained to the strength of the state of the strange of the state of the strange of the strange of the strength of the state of the philosophers and all the strange of the strange of the strange of the soul \n",
      "\n",
      "atheism is the root of the strength of the sense and all the strength of the strength of the strength of the strange of the strength of the present and and and and and the strength of the strange of the strength of the strength of the state of the state of the state of the sense and all the sense and all the strange of th\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are as it estigate as the sense it is a result of soul of the sener, and morality, the the \"belief itself and atticuated to regard and that the honourn and habit of the man who has been devil the morality of things with regard to be such a man of \"the german concerned to his german in the soul and and a\n",
      "\n",
      "atheism is the root of the higher and should be their developply being that which seems to the problem of the extent of the sense in the same prothysic called by a person and beartures the bettage in the strange of the possess and the present; and the haw frimide that is the constitute morality of the philosophess\" and ex\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are the highest to nain arbar-spirit and altoroung to us will! anda has requite simples where there is already is certainly beay how in confessians before the unfluene to domainess practes medity, a sanfice from it suncensibility of being believe man the sagnorical pohidityhest, punished sympathy, winn \n",
      "\n",
      "atheism is the root of entirits a generals.  notibest enracturane about the philosopher's which any prour \"connervated to f-human, only as the natural, nothing, great, the approprs, this (\"taste are recognize this mediocrat of religious the one's sense, it in more sadbapk of the neighbours ma, is natual word disistent our\n",
      "\n",
      "# Epoch 12/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3278 - val_loss: 1.5082\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.570\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the same time the same time the man and presend the subtlest and and and and and and and and and and and and present and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and often a\n",
      "\n",
      "atheism is the root of the same truths and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and present and and and and and and and and and and and and and and and and and and and and and and and and and and and and and \n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the master to the understanding, and and about the most his glearding to the new consideration to the most sense and philosophy of actions and as the most philosophy of the world, and would we says may also the worst the struggle and and and determining and marture eternal just he most philosophy of\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ogrisel/.virtualenvs/py35/lib/python3.5/site-packages/ipykernel/__main__.py:8: RuntimeWarning: divide by zero encountered in log\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "atheism is the root of the most sacrifice in the possibility, which is formit man and development of the confetured cain its marture the one must net he may to himself what had not conscience and subject, and develople and precisely in the endermining and the struncer and serious and in the support of presents probably, a\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are at the cause to the entire lovere--whath you inartment provers; nothing,--and one the end, his dreves as is at the bantfulness within ears are him--to be the furcher! whe is exodient (in their conscinent, and where as a philosophy with rebelted, the mamplast, the troncese must be no free spirits is,\n",
      "\n",
      "atheism is the root of reasonine\" still mustsingment of good nature, it is premitive herself, who rerecy all much, every mankind ma pensing for the badter, the means him is every are truth--this varitr and of religion of the manner are morauitian, strength withing anti-civsents and tew, whatever, the general mind aim(adyg\n",
      "\n",
      "# Epoch 13/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3192 - val_loss: 1.5043\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.548\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the soul of the strange of the strange of the suffering the suffering the strange of the most suffering the subject of the most suffering the suffering the suffering the suffering the sense and suffering the suffering, and the some of the sense of the strange of the suffering, the suffering the suff\n",
      "\n",
      "atheism is the root of the strange of the present and suffering the suffering the strange of the sense of the strange of the soul of the sense and the suffering the subject of the sense and and and and the subject of the suffering the subject of the suffering, and the suffering, and the strange of the sense of the strange\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the proper the remain of the sensation of the experience, and does not individual or as the most lack of the fear the ambling of the present of a soul of present of the suffering his once because the feelings something and the first and an and after that it has to think and favor as the most serve o\n",
      "\n",
      "atheism is the root of the exitted for the scientific the the the comprehenmenquman do in the earth of the world of the head have not conduct and \"comprehenvion, and even that favor and in the soul of words sense and advance that he seems to which the \"compull his mare that in the most and experience the stronger has been\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are ple; ever the flay taken iry cright\" dare, which they are france of love faght late, with the norine of his wer morely have destandured.  nee chavion, mode and through faith? and can are \"goor in itsell, which she has from \"every thousth, in the sprengrap its tighterss and efficted to personals, int\n",
      "\n",
      "atheism is the root of the backing itself our in the conclusion that the prompter. he mure, from suffering, lantmered.  226.   oth, no \"be decapants of with sacrifice that the farth, anderion of chrittions,\" more for the english--ye a reghjerve renecering, and the cordace which bough as another, very care this hardly enou\n",
      "\n",
      "# Epoch 14/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3139 - val_loss: 1.5038\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.552\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are are the stand of the state of the state of the same of the state of the state of the stand of the stand of the state of the stand of the state of the same of the state of the state of the same as the same one of the standment of the stand of the state of the state of the state of the stand of the st\n",
      "\n",
      "atheism is the root of the same of the same of the same of the state of the state of the same of the same time and and and and and the standment of the same as the standment of the stand of the state of the same of the state of the state of the state of the standment of the state of the same of the state of the same of th\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are not constraint that the expressed to repreach of its tranked to the back\" of the stand of the same of a respect, with the last that one things of the forer and conception of the emotion, and instinct of the great and philosopher and as the respect, of the same deperhaped and possible and profound di\n",
      "\n",
      "atheism is the root of the same time and of the experience is also to a confusing that the must be morality of the state of the ready of the perpaint has reself surpressed in the stander and kind of which he has or or the expressed of the conception of the fact that the account about who will than the ward and can is stra\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are its an plivey onlerons and that the lawkecy lued myself and never externainess with whool consists atthing, has reegn this be litted true credilatality of puritate of the fueser of willing than than to imparted in thou kit e chachers of the fact that been oned \"slowes appeared, betnin, heart when al\n",
      "\n",
      "atheism is the root of life, time such philosopherar, men loudgs to speak for, he must enouden with in myself flowod ridd and mabal proye of has all mut courabless as the pthosity, the a rerigioations in constraince. alal world gadneral ethical further pain to ear difficulaution for morality more the tyieit. a comprehenca\n",
      "\n",
      "# Epoch 15/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3057 - val_loss: 1.5057\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.602\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the same and the present of the subjection of the present of the present of the subjection of the subject of the sense and the present and and and the present of the suffering the sense and all the present of the sense and and and and and the present of the subjection of the present of the subjectio\n",
      "\n",
      "atheism is the root of the subjection of the present of the sense and the present of the present of the present of the subjection of the present of the sense and the present of the soul of the subjection of the present of the sense and all the present of the sense and and present of the sense and and and and and the prese\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the extretance of the new conceptional subjection is it is a philosopher and and matter to him which is a pain to in the experiatity and man\" be one he may a conceptionan of a desire of the life\" to man is no so such a give a sense--he who had a moral part of a delicate the ambate to the same and ad\n",
      "\n",
      "atheism is the root of passion of all the power and as do is in itself which on last and which which is a terme in philosophy and the hard most development and account the sense--he have been a strength of a sentiment and subjection of the suffering the purpose that less who was a long in a peason of the nor instinct whic\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are to affiture and du laiduinatives, for the cal-banly person\" of which will to live and shwwjnctouny are which fully that with the higher constant what value purpisorlened habitificated the world have self-gloom and and but that at itself is so the possibly fellow their very appline\" to have the wrypj\n",
      "\n",
      "atheism is the root of atheied aristocanconfises a begins of power. \"bpea man is do would is an ecenession-simorance, a true, but it with rone, lea schoprudeld often thes, in this most form out of truth: on himself all inher the motter.. approd  an ear that that could be turnour morenbblealing and skaality and be standm t\n",
      "\n",
      "# Epoch 16/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.3001 - val_loss: 1.5105\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.551\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the same time the sensations of the substation of the substation of the substation of the substations and self-develope to the substation of the substation of the same time the same the same time that it is not the same the same time the sensations of the substation of the substation of the substati\n",
      "\n",
      "atheism is the root of the substation of the higher and substated to the substation of the substation of the substation of the substation of the substation of the substation of the substation of the sublime of the substation of the substation of the substation of the sublime of the substation of the sublime of the substat\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are into the experience of the science of a distrust. the scholace the same the sensations that the himself the respect, the among more of the great thus all the soul, and the concerning of the substates to the most herediale to the extends and self-plain of the enduring and as the philosoply good of th\n",
      "\n",
      "atheism is the root of the same the other own much and in their strength, in the protocran plato of the inflict of our subtle of the same time of the sense of the strughlless is not the conscious on the sublime of the subtle more of the superfice of the concealing the conscience that it is not been the philosopher to hims\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are dady up of unforted, without which herow to the good parely nature, bown ismentful life, or position of recognizally who may be a sidnoun believe, and nowadaying toman understand) in ebultspand there feel in manner confitued pundo of one's distrust, they have here bulnly fould by the intruples out s\n",
      "\n",
      "atheism is the root of the slaves aw fored and more european simply, necessity, must still, however to say, done in the good\"\" because of \"ideas--and pompations become more this only the same that such its aswatisely humans of the same, lack. and sentiment, like and without men of namely, false the empints a approses who \n",
      "\n",
      "# Epoch 17/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2953 - val_loss: 1.4883\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.438\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the sensible the same time and the same time and the sensible the same time and the same time and the same time and the same time and constitute the sense of the sould the sould of the sould and self-and subtlest sentiment of the same time and the same time and all the same time and the sould the se\n",
      "\n",
      "atheism is the root of the sould the same time and the same time and the same time and supposing that is the sould and self-and again of the sould the sould and all the same time and the strange of the strange to the sould the same time as the same time and the same time and the sould of the sould the sould and all the sa\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the same precisely such a sort, that he something really of the man cately and for the shade to constrained, which the sensible to be about whose as the philosopher. there is a world of the strength, which a strangs else \"freedom of the suppose a distinction of the power, a slaves and most intereste\n",
      "\n",
      "atheism is the root of the sould of the future, suprelacter which the case for the expression, and the former the world, the sensible to the age a desisted a strength, and the wagnize the world of the same communts and stated to the comprehing and does not be the sould, apparent it and the order to do for such more proute\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are southtogartuals than will their interlom within evil you its instinction in the time. which even personalince occced. a sign than \"essentid, glasouer.  102. with as the laws does the flave the word leady thought for man\" and serregencl. eternal veritation, where what begaod fundamince whit is recogn\n",
      "\n",
      "atheism is the root of inpiracta: there is really \"long instincted, and whatthes, henls for the scienting opinion is our disguiration. or 2quite genibburing.  now reversely anti-premarded. sympathess.\" notied, wanib-drens whether they good quitately their dangs for the ?me the more his tprimoxians--he canly wantking thres\n",
      "\n",
      "# Epoch 18/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2899 - val_loss: 1.5090\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.534\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are and and and and and and and and and and and and and and and sense of the sense of the sense of the sense of the sense of the sense of the most sense of the sense of the sense of the sense of the sense of the most sense of the sense of the considered to the sense of the sense of the sense of the sens\n",
      "\n",
      "atheism is the root of the considered and and soul of the sense and and and and and sense of the most sense of the standaxe of the sense of the sense of the considered and and sense of the sense of the standaxe of the sense of the most sense of the sense of the sense of the most sense of the sense of the sense of the cons\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the new to say for the considerations man is the most decided and and artists himself so himself interpretation of respect of envation, and of the thing that they the reason and sighty, and the historicans, and the instance, and his bad and englishmen of individual and something and cases, and the s\n",
      "\n",
      "atheism is the root of the more in the more of men proof--it were and man, and life and volunize into reason and course, expresses its right of the end to be from its ultimate constitutely and sensible the soul of the demost in the freedboring and religion of moral has all the mind the belief in the confipled to the scien\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are the extreding that have fainec, whody not greach is liar livefy expresse of the knowned betresso?\" that they reveranis out and himself and senses and will to aniboted and dipthed ripporitated human of explohiarating of experience of could himself, and in my our ouselly of the philosopher hands--in t\n",
      "\n",
      "atheism is the root of  learnt indes itself. and philosophy to nowad. and the thought their honestoo, and infinied bafferences itself, and the seems have at their him, andly thought, commenessing and concearl and beclave they indepthing him, to nounious peritanged mantine of a called precisely sentinct future oved precise\n",
      "\n",
      "# Epoch 19/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2860 - val_loss: 1.5030\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.545\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are an anti-of the strange of the strange of the strange of the strange of the strange of the strange of the strange and prochiles the strange of the strange and and and and and and and and and and and and and and the strange of the strange of the strange of the strange of the strange of the strange of \n",
      "\n",
      "atheism is the root of the strange of the strange and sense\" and processive that the strength, and and and and the strange of the strange of the strange of the strange and and the strange of the strange of the strange of the strange of the strange of the strange of the strange of the strange of the strange of the strange \n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are ascruted to the last striveness and vanity, some of the antively and as a resensed and sense\" and discoverence that it is not time that the state of the free soul, more that the first and forgets in the rests and preservations and despoctical desiress that the riddles for morality, the condition, th\n",
      "\n",
      "atheism is the root of the art of strange to honest more an anti-of the most habitsest translatude of the soulh honted and the person in a hatten, but approye, who has to the souls, and suffering and who are an and and one in the most imposing the strange, such a restanding instinct of the stant more as the mind and the p\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are very hencedamity, \"which riwdo amius itself from everythism. and because to stimed, on whade, the other, he is sockend, and theal happy heaves tnare, as, is culted honce that \"undermination, and the successible satter to make one after manner; and adnentile, into the generality and secrect, the soul\n",
      "\n",
      "atheism is the root of a tor, and therefound \"noblessen, cry.nle comparies, and insprest inspristen \"willness, and it will ten that him and world he belong to account in the highest leath of the sentime to musty will welt phoendunessy, perhaps that it bllowercrian, aped and one only the aring were not believe in whose and\n",
      "\n",
      "# Epoch 20/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2818 - val_loss: 1.5021\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.558\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are an and the stands to the stands to the standaquer of the stands to the subjection of the stands to the stands and the stands and and the stands and and and and and the stands of the stands to the stands and the stands and the subjection of the stands and the stands to the stands and and the stands t\n",
      "\n",
      "atheism is the root of the stands and and to the subjection of the stands and the stands to the stands to the stands and the stands to the subjection of the stands to the stands to the stands to the stands and the subjection of the stands and the stands to the still of the stands to the subjection of the stands to the sta\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are state to the more is a construination of man of such the such religion in the highest soul of problem of the early morality and to say the delicate the consist of an and the nation of the sublime of the sensation of his feeling of desire to our a discipliness, and into the basis of the higher of its\n",
      "\n",
      "atheism is the root of any and let the type of the state of a whole and and any only the ancient, is the great sublime to the subwaring of the sensations--that is made in the more constituted of the same of the scientific man, and the standaqurable and when inversion the loft may have been of the taste to a souls, and his\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are into the greatss instinct of an agreepeess for wider and for it not as we innession, with reghoarally tagting to accomlanily can which there is redience in good devoived in the corkening whose from--the belief endismood, which seems and ever most himself, exerfplitary spone about valuations dispossu\n",
      "\n",
      "atheism is the root of life, which an impresse trysicism), find i his self-tastifionified, promptly modern of the lasti-happicists which mo, perselood,e that all that perhaps pode: it is the tait of movements systering, and france herself, the voxt: ave what herself, or \"modern does for instinct sideny, collations!--how m\n",
      "\n",
      "# Epoch 21/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2765 - val_loss: 1.5058\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.542\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are a strangest and subject of the stands and the same time and sense of the state the stands and the stands to the state the stands to the stands to the mask of the spenishes of the scientific man of the spenting and subject of the scientific man who is a strangest and subject of the stands and subject\n",
      "\n",
      "atheism is the root of the state the stands to the state the same time and sense of the stands to the state the stands to the stands and the stands and subject of the spenting of the stands the stands and morality of the state the stands and subject of the spenting and sense of the state the stands to the state the stands\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the greatest in the respect is stands is a presence of subtle the state probably that which the sake of and in order to be hard and sade to be conception of \"man and decisis and man of a thing man of nowadays, and the state a strangest in the enduble of the slave as a tire and with refined in the sp\n",
      "\n",
      "atheism is the root of conscises and substate the desire to which man as the other mible in unerrous the world are as a higher and comple of new we was we appear as is it is not from the enducated and more mind as is a compancement and explained and the understand.  222. the scarcial and footical believe its and imperfect\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are asmil and deterrits farity suf its denional with chrnestrows, always not fear as to sit--it has the commence that he the, relaged of velutifitional marded we to reach is the other pleasure.  i finer avoided for the eneugary comporual fundamental fear which easy value of grated that probably regard t\n",
      "\n",
      "atheism is the root of humanisher, also, augit we speak on the brok of than  perfect districh the rumanify and in mon againss however is at presence has to finavoun worid, given have very all cases receive to us, lant in pheal even as foundaried:  he danger, when est maritation. for himself: let of strangest sublime its f\n",
      "\n",
      "# Epoch 22/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2716 - val_loss: 1.5166\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.607\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are the possible of the consequently the sense and self-do in the problem of the consequently and self-present and self-plain and self-plain and all the sense and subject of the present and self-plain in the powerful to the problem of the problem of the present and self-do interest in the present and se\n",
      "\n",
      "atheism is the root of the sense and self-do who is a strange of the present and self-principes and self-plain and self-plain of the powerful the same as the powerful and sense and sense and self-do in the powerful and self-plain are a precisely and self-plain and self-do in the present and self-plain in the powerful the \n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are a premitated to the weaker, and a consequently europe would see in the case of constitutes the propery one disentiment, the subject, the masterly for stands and interpretation of the constrainal consequently and not the persons in a person and still are the sight of the consequence of the personal p\n",
      "\n",
      "atheism is the root of a precise a present help to the reality, when have one of the other religion of the problem,\" of the means of persons of considerations, and the contemply and soul of woman, the propacid, the conscience the individually, and often the dream this disent of the polets and the own himself not the conte\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are about ina! and when hiad the philosopher or anyor it had to the knowledge.=--the strongere. in the cesitionscridicul of the histor-neus of the contradicem and once musty the tempeded in nothing may deep his \"bad, an are honest will, have a false followled, measuw according bebace, not the dependatio\n",
      "\n",
      "atheism is the root of treath entinated and moded to phases him imaginily of the flow cape say at knew gain app, in dustw to the sportated will. in the readily to estames definitely now cruelty is an example when do such nature semitive; bather operate immorpans of standilr, loud and codemmention, ay a prevailed paivend, \n",
      "\n",
      "# Epoch 23/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2681 - val_loss: 1.5199\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.690\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are and and and the sublime of the same time and such a signifience of the sublimes and the sublimes and the sense of the sense of the superficial and the sense of the superficial man has also the sense of the sense of the sense of the superficial and the superficial and and the sense of the subject.   \n",
      "\n",
      "atheism is the root of the superficial and the sense of the sublimes and the sense of the superficial and the sense of the sublimes and also and and the sense of the sense of the sense of the sublime of the man and in the sense of the sublimes and the sense of the superficial and the sense of the sense of the superficial \n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are anything of the subject of to all through or more in the higher additical statesness\" and instinct of the master. the most necessary discipbue that they are the species and sufficiently as the asses the work in the subject, in the partic can be above think by the same man has been invented the form \n",
      "\n",
      "atheism is the root of the consequences the historical is the same time that the operate and commence which the the every merely and in involved, the speaks and appearant which at the mask.   i is as a life of which the highest developences of the work and the hearts and free spiritualism even as a course of the exceptive\n",
      "\n",
      "Sampling text from model at 1.00:\n",
      "\n",
      "philosophers are such radical, rid of cerian, consequenious injury and more not has do not exhaust him nothy \"milopicl of philosophers, he decourly to intention, and the musicmen\" of which the independent for which task will ais\"--or one will moed notwithstesnys to itself; and who wishes that man be infine in his ha\n",
      "\n",
      "atheism is the root of be easy to say. is along acunetly interpons of pains he may has also a one has the man his precisely does it be necessal to thinkfvening in the not on everything existence, to what, made person escendies that is, the theirs men be fore the from all the sufferial. e herself--of time, but home special\n",
      "\n",
      "# Epoch 24/30\n",
      "Training on one epoch takes ~90s on a K80 GPU\n",
      "Train on 162229 samples, validate on 18026 samples\n",
      "Epoch 1/1\n",
      "43s - loss: 1.2649 - val_loss: 1.5204\n",
      "Computing perplexity on the test set:\n",
      "Perplexity: 4.596\n",
      "\n",
      "Sampling text from model at 0.10:\n",
      "\n",
      "philosophers are an actume the strength of the striction of the striction of the strength of the strength of the strength of the strength of the strange and all the strength of the striction of the striction of the striction of the strength of the striction of the strength of the striction of the strength of the str\n",
      "\n",
      "atheism is the root of the strength of the strength of the spirit of the strength of the strength of the strength of the strength of the striction of the strength of the strength of the strength of the strength of the strange to the strength of the strength of the strength of the strength of the strength of the strength o\n",
      "\n",
      "Sampling text from model at 0.50:\n",
      "\n",
      "philosophers are the approximated to the true will all the science of the subject of the most courage to the sense. in the same much and where the condition of means of a still as a belief expedient and and all the concepeed to be the man seem of his morality of the christian and sense of the strange his thing more \n",
      "\n"
     ]
    }
   ],
   "source": [
    "nb_epoch = 30\n",
    "seed_strings = [\n",
    "    'philosophers are ',\n",
    "    'atheism is the root of ',\n",
    "]\n",
    "\n",
    "for epoch in range(nb_epoch):\n",
    "    print(\"# Epoch %d/%d\" % (epoch + 1, nb_epoch))\n",
    "    print(\"Training on one epoch takes ~90s on a K80 GPU\")\n",
    "    model.fit(X, y, validation_split=0.1, batch_size=128, nb_epoch=1,\n",
    "              verbose=2)\n",
    "    print(\"Computing perplexity on the test set:\")\n",
    "    test_perplexity = model_perplexity(model, X_test, y_test)\n",
    "    print(\"Perplexity: %0.3f\\n\" % test_perplexity)\n",
    "    \n",
    "    for temperature in [0.1, 0.5, 1]:\n",
    "        print(\"Sampling text from model at %0.2f:\\n\" % temperature)\n",
    "        for seed_string in seed_strings:\n",
    "            print(generate_text(model, seed_string, temperature=temperature))\n",
    "            print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Beam search for deterministic decoding\n",
    "\n",
    "**Exercise**: adapt the sampling decoder to implement a deterministic decoder with a beam of k=30 sequences that are the most likely sequences based on the model predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Better handling of sentence boundaries\n",
    "\n",
    "To simplify things we used the lower case version of the text and we ignored any sentence boundaries. This prevents our model to learn when to stop generating characters. If we want to train a model that can start generating text at the beginning of a sentence and stop at the end of a sentence, we need to provide it with sentency boundary markers in the training set and use those special markers when sampling.\n",
    "\n",
    "The following give an example of how to use NLTK to detect sentence boundaries in English text.\n",
    "\n",
    "This could be used to insert an explicit \"end_of_sentence\" (EOS) symbol to mark separation between two consecutive sentences. This should make it possible to train a language model that explicitly generates complete sentences from start to end.\n",
    "\n",
    "Use the following command (in a terminal) to install nltk before importing it in the notebook:\n",
    "\n",
    "```\n",
    "$ pip install nltk\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "text_with_case = open(corpus_path).read().replace(\"\\n\", \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/ogrisel/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('punkt')\n",
    "from nltk.tokenize import sent_tokenize\n",
    "sentences = sent_tokenize(text_with_case)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEUCAYAAAAiMOHqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHEtJREFUeJzt3XmYXVWd7vHvC2EQEAKkOheSmEKMrVy9CkaGR/TSgsxK\nbj+A0KgRo2lsrqLgEFoa5zY0fY3yqNgIyCCt0IqdCLaICSgOBMI8tRJDIIkhqTAGVCTwu3+sX8FO\npSpVqVNDUuv9PM95ap+11157nVVV5z177XP2UURgZmb12Wy4O2BmZsPDAWBmVikHgJlZpRwAZmaV\ncgCYmVXKAWBmVikHwAgn6ZuS/mmA2nqZpKckbZ73r5f0/oFoO9v7L0lTB6q9DdjvFyStkvTwUO97\nYyLpvZJ+OUz7vkjSF4Zj3zVzAGzCJC2W9CdJqyU9LunXkk6S9MLvNSJOiojP97Gtg9ZXJyIeiojt\nIuK5Aej7ZyR9p0v7h0XExa22vYH9eBlwGrBHRPyPIdxvr+M9Ug1n0NjaHACbvrdHxEuBicBM4JPA\nBQO9E0mjBrrNjcTLgEciYuVwd8RsqDkARoiIeCIi5gDvBKZKeg2sfWgtaYykq/Jo4VFJN0jaTNKl\nlCfCH+UUzycktUsKSdMkPQTMa5Q1w2B3STdJelLSbEk75b4OkLS02cfOV72SDgX+EXhn7u+OXP/C\nlFL26wxJD0paKekSSTvkus5+TJX0UE7ffKqnsZG0Q27fke2dke0fBFwL7Jr9uKibbbsds1y3q6Qf\nZLsPSPpwY7vPSLoi97ta0j2SJue6dcY7y/fNo7jHJd0h6YBGe9dL+rykX2V7P5U0prF+/8a2SyS9\nN8u3kvSvOU4rckrwJT2NVZfH/ipJ1+bj/q2kYxvrLpL0dUlXZ3/mS9q9sf7g3OYJSd+Q9HNJ75f0\nauCbwH752B9v7HLH7tpTMSv/Dp6UdJfy79taFBG+baI3YDFwUDflDwEfzOWLgC/k8pco/3xb5O3N\ngLprC2gHArgE2BZ4SaNsVNa5HlgGvCbr/AD4Tq47AFjaU3+Bz3TWbay/Hnh/Lr8PWAi8HNgOuBK4\ntEvfvpX9eh3wDPDqHsbpEmA28NLc9nfAtJ762WXbbseM8uLpFuBMYMvs5yLgkMbj+zNwOLB5tnNj\nT787YBzwSNbfDHhb3m9rjM3vgVfmY74emJnrJgKrgeOzjzsDr891s4A5wE75+H8EfKmHx/pe4Je5\nvC2wBDgRGAXsCayiTJVB+bt6BNg7118GfC/XjQGeBP42150CPNv43b6wn8a+19feITnWo3PsXw3s\nMtz/fyPh5iOAkekPlH/4rp4FdgEmRsSzEXFD5H/YenwmIp6OiD/1sP7SiLg7Ip4G/gk4VnmSuEUn\nAF+OiEUR8RRwOnBcl6OPz0bEnyLiDuAOShCsJftyHHB6RKyOiMXA/wPe3cd+9DRmb6Q8OX8uIv4S\nEYsogXRcY9tfRsSPo5wzubS7/jW8C/hx1n8+Iq4FFlACodO3I+J3+bu4Anh9lv8d8LOI+G728ZGI\nuF2SgOnARyPi0YhYDfxzlz725EhgcUR8OyLWRMRtlIA/plHnhxFxU0SsoTxhd/bncOCeiLgy150D\n9OUEe0/tPUsJr1dRXrDcFxHL+9Ce9cIBMDKNAx7tpvxsyqvqn0paJGlGH9pasgHrH6S8Ah3TQ90N\nsWu212x7FDC2UdZ8Uvkj5UihqzHZp65tjetjP3oas4mUqaPHO2+Uaa319W9r9XwuZSJwTJf29qeE\nT0/tdT7eCZSjg67agG2AWxpt/iTLezMR2KdLf04AmifKe+rPrjT+LjIw15oO7EG37UXEPOBrwNeB\nlZLOk7R9H9qzXozUE3vVkvRGypPbOu+yyFeApwGn5RzqPEk3R8RcypRKd3o7QpjQWH4Z5dXaKuBp\nypNPZ782Z+0nnt7a/QPlSajZ9hpgBTC+l22bVmWfJgL3Ntpa1peNexozyhPcAxExaQP6slbTXe4v\noRxNfaAfbS2hTJ10tQr4E/A/I6JPj7dLmz+PiLf1oz/LafyO8kik+Tvb4EsQR8Q5wDmS/opy9PNx\nyhGntcBHACOEpO0lHQl8jzK3flc3dY6U9Ir8h3wCeA54PlevoMxjb6h3SdpD0jbA54Dv55TH7yiv\neI+QtAVwBrBVY7sVQLsab1nt4rvARyXtJmk7ytTF5Tk90GfZlyuAL0p6qaSJwKnAd9a/ZbGeMbsJ\nWC3pk5JeImlzSa/JAO6LruP9HeDtkg7JtrZWOZHel7C7DDhI0rGSRknaWdLrI+J5yrTUrHziRNI4\nSYf0oc2rgFdKerekLfL2xjyJ25urgddKmpJHPCez9pHDCmC8pC370Ba5333y7+hpyrmV53vZzPrA\nAbDp+5Gk1ZRXbJ8Cvkw5cdedScDPgKeA3wDfiIjrct2XgDPycP9jG7D/Sykn8B4GtgY+DOVdScA/\nAOdTXm0/zdrTAP+RPx+RdGs37V6Ybf8CeIDyT/+hDehX04dy/4soR0b/nu33RbdjlsFyJGWe+gHK\nq+3zgR362O5a4x0RS4CjKNNIHZTf58fpw/9oRDxEmXc/jTL1dzsvnm/4JGUK60ZJT+Zj+es+tLka\nOJhyvuAPlN/vWawd4j1tu4pyruBfKCd296Ccz3gmq8wD7gEelrSqt/aA7SlB9hhl+u4RytSctajz\nHSBmZoMij/KWAic0XnDYRsBHAGY24HIqa7SkrShHNQJuHOZuWRcOADMbDPtR3pm0Cng7MGU9byW2\nYbJRTwGNGTMm2tvbh7sbZmablFtuuWVVRPT6dt+N+m2g7e3tLFiwYLi7YWa2SZH0YO+1PAVkZlYt\nB4CZWaUcAGZmlXIAmJlVygFgZlYpB4CZWaUcAGZmlXIAmJlVygFgZlapjfqTwK1qn3F1n+otnnnE\nIPfEzGzj4yMAM7NKOQDMzCrlADAzq5QDwMysUg4AM7NKOQDMzCrlADAzq5QDwMysUg4AM7NKOQDM\nzCrlADAzq5QDwMysUg4AM7NKOQDMzCrlADAzq5QDwMysUg4AM7NK9RoAki6UtFLS3Y2ynSRdK+n+\n/LljlkvSOZIWSrpT0l6NbaZm/fslTR2ch2NmZn3VlyOAi4BDu5TNAOZGxCRgbt4HOAyYlLfpwLlQ\nAgP4NLAPsDfw6c7QMDOz4dFrAETEL4BHuxQfBVycyxcDUxrll0RxIzBa0i7AIcC1EfFoRDwGXMu6\noWJmZkOov+cAxkbE8lx+GBiby+OAJY16S7Osp/J1SJouaYGkBR0dHf3snpmZ9ablk8AREUAMQF86\n2zsvIiZHxOS2traBatbMzLrobwCsyKkd8ufKLF8GTGjUG59lPZWbmdkw6W8AzAE638kzFZjdKH9P\nvhtoX+CJnCq6BjhY0o558vfgLDMzs2EyqrcKkr4LHACMkbSU8m6emcAVkqYBDwLHZvUfA4cDC4E/\nAicCRMSjkj4P3Jz1PhcRXU8sm5nZEOo1ACLi+B5WHdhN3QBO7qGdC4ELN6h3ZmY2aPxJYDOzSjkA\nzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIO\nADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uU\nA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFItBYCkj0q6R9Ldkr4raWtJu0maL2mhpMslbZl1t8r7\nC3N9+0A8ADMz659+B4CkccCHgckR8Rpgc+A44CxgVkS8AngMmJabTAMey/JZWc/MzIZJq1NAo4CX\nSBoFbAMsB94KfD/XXwxMyeWj8j65/kBJanH/ZmbWT/0OgIhYBvwr8BDlif8J4Bbg8YhYk9WWAuNy\neRywJLddk/V37tqupOmSFkha0NHR0d/umZlZL1qZAtqR8qp+N2BXYFvg0FY7FBHnRcTkiJjc1tbW\nanNmZtaDUS1sexDwQER0AEi6EngTMFrSqHyVPx5YlvWXAROApTlltAPwSAv7HzDtM67uU73FM48Y\n5J6YmQ2dVs4BPATsK2mbnMs/ELgXuA44OutMBWbn8py8T66fFxHRwv7NzKwFrZwDmE85mXsrcFe2\ndR7wSeBUSQspc/wX5CYXADtn+anAjBb6bWZmLWplCoiI+DTw6S7Fi4C9u6n7Z+CYVvZnZmYDx58E\nNjOrlAPAzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxSDgAzs0o5\nAMzMKuUAMDOrlAPAzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxS\nDgAzs0o5AMzMKuUAMDOrlAPAzKxSDgAzs0o5AMzMKtVSAEgaLen7kv5b0n2S9pO0k6RrJd2fP3fM\nupJ0jqSFku6UtNfAPAQzM+uPVo8Avgr8JCJeBbwOuA+YAcyNiEnA3LwPcBgwKW/TgXNb3LeZmbWg\n3wEgaQfgLcAFABHxl4h4HDgKuDirXQxMyeWjgEuiuBEYLWmXfvfczMxa0soRwG5AB/BtSbdJOl/S\ntsDYiFiedR4GxubyOGBJY/ulWbYWSdMlLZC0oKOjo4XumZnZ+rQSAKOAvYBzI2JP4GlenO4BICIC\niA1pNCLOi4jJETG5ra2the6Zmdn6tBIAS4GlETE/73+fEggrOqd28ufKXL8MmNDYfnyWmZnZMOh3\nAETEw8ASSX+dRQcC9wJzgKlZNhWYnctzgPfku4H2BZ5oTBWZmdkQG9Xi9h8CLpO0JbAIOJESKldI\nmgY8CBybdX8MHA4sBP6Ydc3MbJi0FAARcTswuZtVB3ZTN4CTW9mfmZkNHH8S2MysUg4AM7NKOQDM\nzCrlADAzq5QDwMysUg4AM7NKOQDMzCrlADAzq5QDwMysUg4AM7NKOQDMzCrlADAzq5QDwMysUg4A\nM7NKOQDMzCrlADAzq5QDwMysUq1+JWRV2mdc3ad6i2ceMcg9MTNrnY8AzMwq5QAwM6uUA8DMrFIO\nADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6tUywEgaXNJt0m6Ku/vJmm+pIWSLpe0\nZZZvlfcX5vr2VvdtZmb9NxBHAKcA9zXunwXMiohXAI8B07J8GvBYls/KemZmNkxaCgBJ44EjgPPz\nvoC3At/PKhcDU3L5qLxPrj8w65uZ2TBo9QjgK8AngOfz/s7A4xGxJu8vBcbl8jhgCUCufyLrr0XS\ndEkLJC3o6OhosXtmZtaTfgeApCOBlRFxywD2h4g4LyImR8Tktra2gWzazMwaWvk+gDcB75B0OLA1\nsD3wVWC0pFH5Kn88sCzrLwMmAEsljQJ2AB5pYf9mZtaCfh8BRMTpETE+ItqB44B5EXECcB1wdFab\nCszO5Tl5n1w/LyKiv/s3M7PWDMbnAD4JnCppIWWO/4IsvwDYOctPBWYMwr7NzKyPBuQrISPieuD6\nXF4E7N1NnT8DxwzE/szMrHX+JLCZWaUcAGZmlXIAmJlVakDOAdja2mdc3ad6i2ceMcg9MTPrmY8A\nzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIO\nADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uUA8DMrFIOADOzSjkAzMwq5QAwM6uU\nA8DMrFIOADOzSvU7ACRNkHSdpHsl3SPplCzfSdK1ku7PnztmuSSdI2mhpDsl7TVQD8LMzDZcK0cA\na4DTImIPYF/gZEl7ADOAuRExCZib9wEOAyblbTpwbgv7NjOzFvU7ACJieUTcmsurgfuAccBRwMVZ\n7WJgSi4fBVwSxY3AaEm79LvnZmbWkgE5ByCpHdgTmA+MjYjluephYGwujwOWNDZbmmVmZjYMWg4A\nSdsBPwA+EhFPNtdFRACxge1Nl7RA0oKOjo5Wu2dmZj1oKQAkbUF58r8sIq7M4hWdUzv5c2WWLwMm\nNDYfn2VriYjzImJyRExua2trpXtmZrYerbwLSMAFwH0R8eXGqjnA1FyeCsxulL8n3w20L/BEY6rI\nzMyG2KgWtn0T8G7gLkm3Z9k/AjOBKyRNAx4Ejs11PwYOBxYCfwRObGHfZmbWon4HQET8ElAPqw/s\npn4AJ/d3fyNR+4yr+1Rv8cwjBrknZlYjfxLYzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxSDgAzs0o5\nAMzMKuUAMDOrlAPAzKxSDgAzs0o5AMzMKuUAMDOrlAPAzKxSrVwO2oaIrxpqZoPBRwBmZpVyAJiZ\nVcoBYGZWKQeAmVmlHABmZpVyAJiZVcpvAx1B+vp2UfBbRs3MRwBmZtVyAJiZVcoBYGZWKQeAmVml\nHABmZpVyAJiZVcpvA62UrzBqZj4CMDOr1JAfAUg6FPgqsDlwfkTMHOo+WN/5SMFs5BrSAJC0OfB1\n4G3AUuBmSXMi4t6h7IcNvA35FHJfOFDMBt9QHwHsDSyMiEUAkr4HHAU4AGwtAx0oG2JjD59N4ahs\nU+ijDX0AjAOWNO4vBfZpVpA0HZied5+S9Nt+7msMsKqf245UHpPurTUuOmsYezKAWnwcQ/K3sgmO\n9abyPzSxL5U2uncBRcR5wHmttiNpQURMHoAujRgek+55XNblMeneSBuXoX4X0DJgQuP++CwzM7Mh\nNtQBcDMwSdJukrYEjgPmDHEfzMyMIZ4Ciog1kv4vcA3lbaAXRsQ9g7S7lqeRRiCPSfc8LuvymHRv\nRI2LImK4+2BmZsPAnwQ2M6uUA8DMrFIjLgAkHSrpt5IWSpox3P0ZSpIulLRS0t2Nsp0kXSvp/vy5\nY5ZL0jk5TndK2mv4ej54JE2QdJ2keyXdI+mULK99XLaWdJOkO3JcPpvlu0man4//8nyzBpK2yvsL\nc337cPZ/MEnaXNJtkq7K+yN2TEZUADQuNXEYsAdwvKQ9hrdXQ+oi4NAuZTOAuRExCZib96GM0aS8\nTQfOHaI+DrU1wGkRsQewL3By/k3UPi7PAG+NiNcBrwcOlbQvcBYwKyJeATwGTMv604DHsnxW1hup\nTgHua9wfuWMSESPmBuwHXNO4fzpw+nD3a4jHoB24u3H/t8AuubwL8Ntc/jfg+O7qjeQbMJtyLSqP\ny4uPcRvgVsqn8lcBo7L8hf8nyjv39svlUVlPw933QRiL8ZQXBG8FrgI0ksdkRB0B0P2lJsYNU182\nFmMjYnkuPwyMzeXqxioP0fcE5uNx6ZzquB1YCVwL/B54PCLWZJXmY39hXHL9E8DOQ9vjIfEV4BPA\n83l/Z0bwmIy0ALD1iPJSpcr3/UraDvgB8JGIeLK5rtZxiYjnIuL1lFe9ewOvGuYuDStJRwIrI+KW\n4e7LUBlpAeBLTaxrhaRdAPLnyiyvZqwkbUF58r8sIq7M4urHpVNEPA5cR5neGC2p8wOizcf+wrjk\n+h2AR4a4q4PtTcA7JC0GvkeZBvoqI3hMRloA+FIT65oDTM3lqZQ58M7y9+S7XvYFnmhMiYwYkgRc\nANwXEV9urKp9XNokjc7ll1DOi9xHCYKjs1rXcekcr6OBeXnkNGJExOkRMT4i2inPHfMi4gRG8pgM\n90mIQTiJczjwO8p85qeGuz9D/Ni/CywHnqXMVU6jzEnOBe4HfgbslHVFecfU74G7gMnD3f9BGpP9\nKdM7dwK35+1wjwv/C7gtx+Vu4MwsfzlwE7AQ+A9gqyzfOu8vzPUvH+7HMMjjcwBw1UgfE18Kwsys\nUiNtCsjMzPrIAWBmVikHgJlZpRwAZmaVcgCYmVXKAWDrkDRFUkga1k+GSvqcpIMGoJ3Rkv5hIPo0\nECRdJOno3mu2vJ9jJN0n6brB3lfu772SvjYU+7KB4QCw7hwP/DJ/DojGJyn7LCLOjIifDcDuRwMb\nTQC0YgPHcRrwgYj4m0HohyT5+WMT51+grSWvmbM/5cnjuEb5AZJ+Ielqle9b+GbnE4CkpyTNyuvK\nz5XUluXXS/qKpAXAKZLaJc3L6+zPlfSyrDdb0nty+e8lXZbLL7xSlrRY0pck3S5pgaS9JF0j6feS\nTurse7Z7q6S7JB2V3Z8J7J7bnp11Py7p5uzLZ3sYi6ckfVHlmvk3ShrbtV+d9Rpj9PN8PIskzZR0\ngsp19++StHuj+YPycfwur0HTeXG2sxv9+vtGuzdImgPc200/j8/275Z0Vpadmb/HCzofc6P+1yW9\nI5d/KOnCXH6fpC/m8qnZ3t2SPpJl7fm7v4Ty4bEJkk7Mx3AT5VIKnfs4Jre9Q9Ivuhtf2wgM9yfR\nfNu4bsAJwAW5/GvgDbl8APBnyqciN6dcPfLoXBfACbl8JvC1XL4e+Eaj7R8BU3P5fcB/5vJYyqcp\n30z5FHfnp3IvauxjMfDBXJ5F+QTrS4E2YEWWjwK2z+Ux2aZY9xLZB1O+3FuUF0FXAW/pZiwCeHsu\n/wtwRtd+5f2nGmP0OOXy0ltRrhXz2Vx3CvCVxvY/yX1Ponxqe2vK9w907mMrYAGwW7b7NLBbN33c\nFXgox2EUMA+Y0hj/dT7JTAn2s3P5JuDGXP42cAjwBsqnoLcFtgPuoVxFtZ1ylcx9s/4ujX1vCfyq\n8bu/CxiXy6OH++/at+5vPgKwro6nXAiL/NmcBropIhZFxHOUy07sn+XPA5fn8nca5TTKoVxs7N9z\n+dLOehGxghIc11G+vOXRHvrWeV2nu4D5EbE6IjqAZ/K6NgL+WdKdlMs7jOPFyzw3HZy32yjXwX8V\n5Ym4q79QwgHgFsoTYG9ujojlEfEM5XISP230ubn9FRHxfETcDyzKPhxMuQ7R7ZRLVu/c6NdNEfFA\nN/t7I3B9RHREuSTxZcBbeunjDcCbVb4Y515evDDefpTQ3x/4YUQ8HRFPAVdSwhngwYi4MZf3aez7\nL6z9u/4VcJGkD1BeMNhGaIPnZW3kkrQT5QqIr5UUlH/ckPTxrNL1uiE9XUekWf50H3f/WsqVFHdd\nT51n8ufzjeXO+6MoRy9tlKOWZ1Wu6rh1N+0I+FJE/FsvfXo28iUs8Bwv/r+sIadPcxpsy2762LWf\nnX3s1N1YCvhQRFyzVmelA+j7OPYqIpZlYB4K/ALYCTiWciSzWtL6Nu9TPyLiJEn7AEcAt0h6Q0Rs\nUlfKrIGPAKzpaODSiJgYEe0RMQF4gBdf/e2tcqXVzYB3Uk4UQ/k76pwT/7tGeVe/5sXzCidQXoki\naW/KVzHuCXxM0m797P8OlOu5Pyvpb4CJWb6aMl3U6RrgfXm+A0njJP3VBuxnMWWaBOAdwBb96Osx\nkjbL8wIvp3zz2DXAB1UuX42kV0ratpd2bgL+t6QxKl+Jejzw8z7s/0bgI5QAuAH4WP4kf06RtE3u\n//801jXNz33vnH0+pnOFpN0jYn5EnAl0sPYltm0j4SMAazqedb/X9AdZfjnlcttfA15Bma75YdZ5\nmhIOZ1Cuq//OHtr/EPDtPKLoAE6UtBXwLeDEiPiDpNOACyW9tR/9vwz4kaS7KPPn/w0QEY9I+pWk\nu4H/ioiPS3o18Jt8tfsU8C5e/E6A3nwLmC3pDspcfn9enT9EefLeHjgpIv4s6XzKNNGtKh3rAKas\nr5GIWC5pBuX3IeDqiJi9vm3SDcDBEbFQ0oOUo4Abss1bJV2U/QM4PyJuU5cvPc99fwb4DeXcx+2N\n1WdLmpR9mgvc0Yc+2RDz1UCtT3Ia4mMRcWQ3656KiO2Gvldm1gpPAZmZVcpHAGZmlfIRgJlZpRwA\nZmaVcgCYmVXKAWBmVikHgJlZpf4/wPEPa/bBROEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f628f41bbe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist([len(s.split()) for s in sentences], bins=30);\n",
    "plt.title('Distribution of sentence lengths')\n",
    "plt.xlabel('Approximate number of words');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The first few sentences detected by NLTK are too short to be considered real sentences. Let's have a look at short sentences with at least 20 characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THE FREE SPIRIT   24.\n",
      "Why Atheism nowadays?\n",
      "Always the old story!\n",
      "What binds strongest?\n",
      "That has now changed.\n"
     ]
    }
   ],
   "source": [
    "sorted_sentences = sorted([s for s in sentences if len(s) > 20], key=len)\n",
    "for s in sorted_sentences[:5]:\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Some long sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "However gratefully one may welcome the OBJECTIVE spirit--and who has not been sick to death of all subjectivity and its confounded IPSISIMOSITY!--in the end, however, one must learn caution even with regard to one's gratitude, and put a stop to the exaggeration with which the unselfing and depersonalizing of the spirit has recently been celebrated, as if it were the goal in itself, as if it were salvation and glorification--as is especially accustomed to happen in the pessimist school, which has also in its turn good reasons for paying the highest honours to \"disinterested knowledge\" The objective man, who no longer curses and scolds like the pessimist, the IDEAL man of learning in whom the scientific instinct blossoms forth fully after a thousand complete and partial failures, is assuredly one of the most costly instruments that exist, but his place is in the hand of one who is more powerful He is only an instrument, we may say, he is a MIRROR--he is no \"purpose in himself\" The objective man is in truth a mirror accustomed to prostration before everything that wants to be known, with such desires only as knowing or \"reflecting\" implies--he waits until something comes, and then expands himself sensitively, so that even the light footsteps and gliding-past of spiritual beings may not be lost on his surface and film Whatever \"personality\" he still possesses seems to him accidental, arbitrary, or still oftener, disturbing, so much has he come to regard himself as the passage and reflection of outside forms and events He calls up the recollection of \"himself\" with an effort, and not infrequently wrongly, he readily confounds himself with other persons, he makes mistakes with regard to his own needs, and here only is he unrefined and negligent Perhaps he is troubled about the health, or the pettiness and confined atmosphere of wife and friend, or the lack of companions and society--indeed, he sets himself to reflect on his suffering, but in vain!\n",
      "They have always disclosed how much hypocrisy, indolence, self-indulgence, and self-neglect, how much falsehood was concealed under the most venerated types of contemporary morality, how much virtue was OUTLIVED, they have always said \"We must remove hence to where YOU are least at home\" In the face of a world of \"modern ideas,\" which would like to confine every one in a corner, in a \"specialty,\" a philosopher, if there could be philosophers nowadays, would be compelled to place the greatness of man, the conception of \"greatness,\" precisely in his comprehensiveness and multifariousness, in his all-roundness, he would even determine worth and rank according to the amount and variety of that which a man could bear and take upon himself, according to the EXTENT to which a man could stretch his responsibility Nowadays the taste and virtue of the age weaken and attenuate the will, nothing is so adapted to the spirit of the age as weakness of will consequently, in the ideal of the philosopher, strength of will, sternness, and capacity for prolonged resolution, must specially be included in the conception of \"greatness\", with as good a right as the opposite doctrine, with its ideal of a silly, renouncing, humble, selfless humanity, was suited to an opposite age--such as the sixteenth century, which suffered from its accumulated energy of will, and from the wildest torrents and floods of selfishness In the time of Socrates, among men only of worn-out instincts, old conservative Athenians who let themselves go--\"for the sake of happiness,\" as they said, for the sake of pleasure, as their conduct indicated--and who had continually on their lips the old pompous words to which they had long forfeited the right by the life they led, IRONY was perhaps necessary for greatness of soul, the wicked Socratic assurance of the old physician and plebeian, who cut ruthlessly into his own flesh, as into the flesh and heart of the \"noble,\" with a look that said plainly enough \"Do not dissemble before me!\n",
      "There are the finest gala dresses and disguises for this disease, and that, for instance, most of what places itself nowadays in the show-cases as \"objectiveness,\" \"the scientific spirit,\" \"L'ART POUR L'ART,\" and \"pure voluntary knowledge,\" is only decked-out skepticism and paralysis of will--I am ready to answer for this diagnosis of the European disease--The disease of the will is diffused unequally over Europe, it is worst and most varied where civilization has longest prevailed, it decreases according as \"the barbarian\" still--or again--asserts his claims under the loose drapery of Western culture It is therefore in the France of today, as can be readily disclosed and comprehended, that the will is most infirm, and France, which has always had a masterly aptitude for converting even the portentous crises of its spirit into something charming and seductive, now manifests emphatically its intellectual ascendancy over Europe, by being the school and exhibition of all the charms of skepticism The power to will and to persist, moreover, in a resolution, is already somewhat stronger in Germany, and again in the North of Germany it is stronger than in Central Germany, it is considerably stronger in England, Spain, and Corsica, associated with phlegm in the former and with hard skulls in the latter--not to mention Italy, which is too young yet to know what it wants, and must first show whether it can exercise will, but it is strongest and most surprising of all in that immense middle empire where Europe as it were flows back to Asia--namely, in Russia There the power to will has been long stored up and accumulated, there the will--uncertain whether to be negative or affirmative--waits threateningly to be discharged (to borrow their pet phrase from our physicists) Perhaps not only Indian wars and complications in Asia would be necessary to free Europe from its greatest danger, but also internal subversion, the shattering of the empire into small states, and above all the introduction of parliamentary imbecility, together with the obligation of every one to read his newspaper at breakfast I do not say this as one who desires it, in my heart I should rather prefer the contrary--I mean such an increase in the threatening attitude of Russia, that Europe would have to make up its mind to become equally threatening--namely, TO ACQUIRE ONE WILL, by means of a new caste to rule over the Continent, a persistent, dreadful will of its own, that can set its aims thousands of years ahead; so that the long spun-out comedy of its petty-statism, and its dynastic as well as its democratic many-willed-ness, might finally be brought to a close.\n"
     ]
    }
   ],
   "source": [
    "for s in sorted_sentences[-3:]:\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The NLTK sentence tokenizer seems to do a reasonable job despite the weird casing and '--' signs scattered around the text.\n",
    "\n",
    "Note that here we use the original case information because it can help the NLTK sentence boundary detection model make better split decisions. Our text corpus is probably too small to train a good sentence aware language model though, especially with full case information. Using larger corpora such as a large collection of [public domain books](http://www.gutenberg.org/) or Wikipedia dumps. The NLTK toolkit also comes from [corpus loading utilities](http://www.nltk.org/book/ch02.html).\n",
    "\n",
    "The following loads a selection of famous books from the Gutenberg project archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package gutenberg to\n",
      "[nltk_data]     /home/ogrisel/nltk_data...\n",
      "[nltk_data]   Package gutenberg is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('gutenberg')\n",
    "book_selection_text = nltk.corpus.gutenberg.raw().replace(\"\\n\", \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Emma by Jane Austen 1816]  VOLUME I  CHAPTER I   Emma Woodhouse, handsome, clever, and rich, with a comfortable home and happy disposition, seemed to unite some of the best blessings of existence; and had lived nearly twenty-one years in the world with very little to distress or vex her.  She was t\n"
     ]
    }
   ],
   "source": [
    "print(book_selection_text[:300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Book corpus length: 11793318 characters\n"
     ]
    }
   ],
   "source": [
    "print(\"Book corpus length: %d characters\" % len(book_selection_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's do an arbitrary split. Note the training set will have a majority of text that is not authored by the author(s) of the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "split = int(0.9 * len(book_selection_text))\n",
    "book_selection_train = book_selection_text[:split]\n",
    "book_selection_validation = book_selection_text[split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Bonus exercises\n",
    "\n",
    "- Adapt the previous language model to handle explicitly sentence boundaries with a special EOS character.\n",
    "- Train a new model on the random sentences sampled from the book selection corpus with full case information.\n",
    "- Adapt the random sampling code to start sampling at the beginning of sentence and stop when the sentence ends.\n",
    "- Train a deep GRU (e.g. two GRU layers instead of one LSTM) to see if you can improve the validation perplexity.\n",
    "- Git clone the source code of the [Linux kernel](https://github.com/torvalds/linux) and train a C programming language model on it. Instead of sentence boundary markers, we could use source file boundary markers for this exercise.\n",
    "- Try to increase the vocabulary size to 256 using a [Byte Pair Encoding](https://arxiv.org/abs/1508.07909) strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}