21_Machine_Translation_CN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #21\n",
    "# 机器翻译\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)/[GitHub中文](https://github.com/Hvass-Labs/TensorFlow-Tutorials-Chinese)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)\n",
    "\n",
    "中文翻译[ZhouGeorge](https://github.com/ZhouGeorge)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 介绍\n",
    "\n",
    "教程 #20 展示了如何运用循环神经网络（RNN）去完成电影评论的情感分析。这份教程通过两个神经网络合并将这个想法扩展到人类语言的机器翻译。\n",
    "\n",
    "\n",
    "你需要要熟悉TensorFlow，Keras和自然语言处理的基础，详见 #01 ， #03-C和 #20。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 流程图\n",
    "\n",
    "下面的流程图大致展示了神经网络是如何构造的。它被分为两个部分：一个编码器将源文本映射到“思维向量”（thought vector），它总结了文本的内容，然后将它输入到神经网络的第二个部分，将“思维向量”（thought vector）解码成目标文本。\n",
    "\n",
    "神经网络不能直接作用到文本上，所以我们首先要用分词器将每一个词转换到整形代号（integer-token）。但是神经网络也不能直接作用在整数上，所以我们要用Embedding层将整形代号（integer-token）转换成浮点形的向量。embedding层与神经网络的其他部分一起训练，主要用于将相似情感的词映射到相似的浮点向量上。\n",
    "\n",
    "举个例子，丹麦语 \"der var engang\"，是每一个通话开始的语句，字面的翻译是\"there was once\"，但是它通常被翻译成英语\"once upon a time\"。我们首先将全部的数据集转换成整形代号，所以 \"der var engang\" 变成了 [12, 54, 1097]。每一个整形代号（integer-tokens）被映射到一个嵌入向量（embedding-vector），例如有128个元素的，所以整形代号12可以变成例如[0.12, -0.56, ..., 1.19]，整形代号54可以变成例如 [0.39, 0.09, ..., -0.12].，这些嵌入（ embedding-vectors）向量之后可以被输入到有3个GRU层的循环神经网络。更多的细节见 #20.\n",
    "\n",
    "最后一个GRU层输出单独的向量-即总结了原文本内容的“思维向量”-它然后作为解码部分中GUR单元的初始状态。\n",
    "\n",
    "目标文本\"once upon a time\"被特殊的记号“ssss”和“eeee”填充，意味着文本的开始和结尾，所以整形代号的序列变成了[2, 337, 640, 9, 79, 3].。在训练过程中，解码器会以全部的序列作为输入，期望的输出序列是 [337, 640, 9, 79, 3] ，与上面的序列相同但时间偏移了一步。我们尝试教编码器去映射“思维向量”和起始代号\"ssss\"（整数2）到下一个词\"once\"（整形 337），然后将 \"once\"映射到\"upon\"（整数 640），诸如此类。\n",
    "\n",
    "\n",
    "下面的流程图描述了主要的思想，但是没有展示出全部必须的细节，例如关于损失函数内容，它也有点复杂。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Flowchart](images/21_machine_translation_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要从Keras中导入一些模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Model  # This does not work!\n",
    "from tensorflow.python.keras.models import Model\n",
    "from tensorflow.python.keras.layers import Input, Dense, GRU, Embedding\n",
    "from tensorflow.python.keras.optimizers import RMSprop\n",
    "from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard\n",
    "from tensorflow.python.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.python.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "开发环境Python 3.6，等等："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.5.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.2-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据\n",
    "\n",
    "我将使用欧洲议会数据集，包含了大多数欧洲语言的句子对。这些数据是由欧盟创建的，将大量的通信翻译成欧盟成员国的语言。\n",
    "\n",
    "http://www.statmt.org/europarl/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import europarl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这份教程中我们使用了英语-丹麦语（English-Danish）的数据集，它包含了2百万个句子对。你可以通过改变language-code来使用另一种语言，具体见`europarl.py`中关于可用语言代码的列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "language_code='da'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了让解码器知道什么是一个序列的开始和结束，我们需要用不可能出现在数据集中的单词来标记每一个序列的开始和结束。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "mark_start = 'ssss '\n",
    "mark_end = ' eeee'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以根据喜好来改变数据文件的地址。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_dir = \"data/europarl/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你没有准备好数据，它会被自动下载和解压。\n",
    "\n",
    "**注意 :English-Danish数据集大约有587 MB！**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has apparently already been downloaded and unpacked.\n"
     ]
    }
   ],
   "source": [
    "europarl.maybe_download_and_extract(language_code=language_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载源语言的文本，这里使用丹麦语。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_src = europarl.load_data(english=False,\n",
    "                              language_code=language_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "记载目标语言的文件，这里是英语。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dest = europarl.load_data(english=True,\n",
    "                               language_code=language_code,\n",
    "                               start=mark_start,\n",
    "                               end=mark_end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将建立一个将源语言（丹麦语）翻译到目标语言（英语）的模型。如果你想做一个逆向的翻译，你仅仅需要改变源数据和目标数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 实例数据\n",
    "\n",
    "数据是一列排序好的文本，源文文本和目标文本的索引是相匹配的。我们可以找个例子确认下翻译是否对应。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Som De kan se, indfandt det store \"år 2000-problem\" sig ikke. Til gengæld har borgerne i en del af medlemslandene været ramt af meget forfærdelige naturkatastrofer.'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"ssss Although, as you will have seen, the dreaded 'millennium bug' failed to materialise, still the people in a number of countries suffered a series of natural disasters that truly were dreadful. eeee\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  数据中的错误\n",
    "\n",
    "数据集中包括2百万个句子对。有些数据是不正确的。这个例子中出现了法语（或者是一些奇怪的语言），尽管里面也有丹麦语。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 8002"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"Car il savait ce que cette foule en joie ignorait, et qu\\'on peut lire dans les livres, que le bacille de la peste ne meurt ni ne disparaît jamais, qu\\'il peut rester pendant des dizaines d\\'années endormi dans les meubles et le linge, qu\\'il attend patiemment dans les chambres, les caves, les malles, les mouchoirs et les paperasses, et que, peut-être, le jour viendrait où, pour le malheur et l\\'enseignement des hommes, la peste réveillerait ses rats et les enverrait mourir dans une cité heureuse.\" (Thi han vidste det, som denne glade forsamling ikke vidste, og som man kan læse i bøger, at pestens bacille aldrig dør og aldrig forsvinder, at den kan sove i mange år i møbler og linned, at den venter tålmodigt i kamre, kældre, kufferter, lommetørklæder og papirer, og at den dag måske kommer, hvor pesten til menneskenes skade og oplysning vågner sine rotter og sender dem ud for at dø i en lykkelig by.)'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss \"He knew what those jubilant crowds did not know but could have learned from books: that the plague bacillus never dies or disappears for good; that it can lie dormant for years and years in furniture and linen-chests; that it bides its time in bedrooms, cellars, trunks, and bookshelves; and that perhaps the day would come when, for the bane and the enlightening of men, it would rouse up its rats again and send them forth to die in a happy city.\" eeee'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分词器\n",
    "\n",
    "神经网络不能直接作用到文本。我们需要用两步处理，将文本转换成可以出入神经网络的数字。第一步是将文本词汇转换成整形代号（integer-tokens）。第二步是利用embedding层将整形代号转换成浮点值的向量。更多的细节见教程 #20.\n",
    "\n",
    "设置词汇表的最大词汇数。这意味着我们将用数据集中（例如）10000个频率最高的词汇。这里我们对源语言和目标语言中设置了相同的大小，但这其实也是可以设置成不同的大小。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要的功能比Keras中分词器类能提供的功能更多，所以我们将它们封装起来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TokenizerWrap(Tokenizer):\n",
    "    \"\"\"Wrap the Tokenizer-class from Keras with more functionality.\"\"\"\n",
    "    \n",
    "    def __init__(self, texts, padding,\n",
    "                 reverse=False, num_words=None):\n",
    "        \"\"\"\n",
    "        :param texts: List of strings. This is the data-set.\n",
    "        :param padding: Either 'post' or 'pre' padding.\n",
    "        :param reverse: Boolean whether to reverse token-lists.\n",
    "        :param num_words: Max number of words to use.\n",
    "        \"\"\"\n",
    "\n",
    "        Tokenizer.__init__(self, num_words=num_words)\n",
    "\n",
    "        # Create the vocabulary from the texts.\n",
    "        self.fit_on_texts(texts)\n",
    "\n",
    "        # Create inverse lookup from integer-tokens to words.\n",
    "        self.index_to_word = dict(zip(self.word_index.values(),\n",
    "                                      self.word_index.keys()))\n",
    "\n",
    "        # Convert all texts to lists of integer-tokens.\n",
    "        # Note that the sequences may have different lengths.\n",
    "        self.tokens = self.texts_to_sequences(texts)\n",
    "\n",
    "        if reverse:\n",
    "            # Reverse the token-sequences.\n",
    "            self.tokens = [list(reversed(x)) for x in self.tokens]\n",
    "        \n",
    "            # Sequences that are too long should now be truncated\n",
    "            # at the beginning, which corresponds to the end of\n",
    "            # the original sequences.\n",
    "            truncating = 'pre'\n",
    "        else:\n",
    "            # Sequences that are too long should be truncated\n",
    "            # at the end.\n",
    "            truncating = 'post'\n",
    "\n",
    "        # The number of integer-tokens in each sequence.\n",
    "        self.num_tokens = [len(x) for x in self.tokens]\n",
    "\n",
    "        # Max number of tokens to use in all sequences.\n",
    "        # We will pad / truncate all sequences to this length.\n",
    "        # This is a compromise so we save a lot of memory and\n",
    "        # only have to truncate maybe 5% of all the sequences.\n",
    "        self.max_tokens = np.mean(self.num_tokens) \\\n",
    "                          + 2 * np.std(self.num_tokens)\n",
    "        self.max_tokens = int(self.max_tokens)\n",
    "\n",
    "        # Pad / truncate all token-sequences to the given length.\n",
    "        # This creates a 2-dim numpy matrix that is easier to use.\n",
    "        self.tokens_padded = pad_sequences(self.tokens,\n",
    "                                           maxlen=self.max_tokens,\n",
    "                                           padding=padding,\n",
    "                                           truncating=truncating)\n",
    "\n",
    "    def token_to_word(self, token):\n",
    "        \"\"\"Lookup a single word from an integer-token.\"\"\"\n",
    "\n",
    "        word = \" \" if token == 0 else self.index_to_word[token]\n",
    "        return word \n",
    "\n",
    "    def tokens_to_string(self, tokens):\n",
    "        \"\"\"Convert a list of integer-tokens to a string.\"\"\"\n",
    "\n",
    "        # Create a list of the individual words.\n",
    "        words = [self.index_to_word[token]\n",
    "                 for token in tokens\n",
    "                 if token != 0]\n",
    "        \n",
    "        # Concatenate the words to a single string\n",
    "        # with space between all the words.\n",
    "        text = \" \".join(words)\n",
    "\n",
    "        return text\n",
    "    \n",
    "    def text_to_tokens(self, text, reverse=False, padding=False):\n",
    "        \"\"\"\n",
    "        Convert a single text-string to tokens with optional\n",
    "        reversal and padding.\n",
    "        \"\"\"\n",
    "\n",
    "        # Convert to tokens. Note that we assume there is only\n",
    "        # a single text-string so we wrap it in a list.\n",
    "        tokens = self.texts_to_sequences([text])\n",
    "        tokens = np.array(tokens)\n",
    "\n",
    "        if reverse:\n",
    "            # Reverse the tokens.\n",
    "            tokens = np.flip(tokens, axis=1)\n",
    "\n",
    "            # Sequences that are too long should now be truncated\n",
    "            # at the beginning, which corresponds to the end of\n",
    "            # the original sequences.\n",
    "            truncating = 'pre'\n",
    "        else:\n",
    "            # Sequences that are too long should be truncated\n",
    "            # at the end.\n",
    "            truncating = 'post'\n",
    "\n",
    "        if padding:\n",
    "            # Pad and truncate sequences to the given length.\n",
    "            tokens = pad_sequences(tokens,\n",
    "                                   maxlen=self.max_tokens,\n",
    "                                   padding='pre',\n",
    "                                   truncating=truncating)\n",
    "\n",
    "        return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以创建源语言的分词器。注意，我们选择('pre')来对序列填充0。我们也反转了序列的代号，研究表明这样可以提高性能，因为编码器所看到的最后一个词与解码器产生的第一个词相匹配，因此短期依赖性被认为能更精确地建模。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 17s, sys: 608 ms, total: 2min 17s\n",
      "Wall time: 2min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer_src = TokenizerWrap(texts=data_src,\n",
    "                              padding='pre',\n",
    "                              reverse=True,\n",
    "                              num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在可以创建目标语言的分词器。我们对源语言和目标语言都需要创建分词器是因为它们的词汇表是不同的。注意分词器不会反转序列，它会在数组的末尾('post')填0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 42s, sys: 492 ms, total: 1min 42s\n",
      "Wall time: 1min 42s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer_dest = TokenizerWrap(texts=data_dest,\n",
    "                               padding='post',\n",
    "                               reverse=False,\n",
    "                               num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为填充过的代号序列定义变量。这些只是是2维的整形代号（integer-tokens）的numpy数组\n",
    "\n",
    "注意，对于源语言和目标语言，序列的长度是不同。这是因为具有相同含义的文本在这两种语言中可能有不同数量的单词。\n",
    "\n",
    "而且，我们在对原始文本分词时为了节约内存做了一个妥协。这意味着我们只截取了大约5%的文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1968800, 47)\n",
      "(1968800, 55)\n"
     ]
    }
   ],
   "source": [
    "tokens_src = tokenizer_src.tokens_padded\n",
    "tokens_dest = tokenizer_dest.tokens_padded\n",
    "print(tokens_src.shape)\n",
    "print(tokens_dest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在目标语言中，用于标记文本开始的标记整形代号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_start = tokenizer_dest.word_index[mark_start.strip()]\n",
    "token_start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在目标语言中，用于标记文本结束的标记整形代号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_end = tokenizer_dest.word_index[mark_end.strip()]\n",
    "token_end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 代号序列的例子"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是分词器的输出，注意，在序列的开始先用0填充了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0, 3069,\n",
       "       3374,   43,    7, 1386,  108, 1995,    7,  178,    9,    3,  302,\n",
       "         19, 2076,    8,   20,   39,  285,  499,   69,  136,    5,  166,\n",
       "         24,   10,   13], dtype=int32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_src[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过整形代号转换到相应的单词来重构原始的文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'naturkatastrofer forfærdelige meget af ramt været medlemslandene af del en i borgerne har gengæld til ikke sig problem 2000 år store det se kan de som'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_src.tokens_to_string(tokens_src[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个文本其实已经被反转了，数据集中原始的文本如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Som De kan se, indfandt det store \"år 2000-problem\" sig ikke. Til gengæld har borgerne i en del af medlemslandene været ramt af meget forfærdelige naturkatastrofer.'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_src[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是目标语言的对应文本的的整形代号序列。注意它的末尾被0填充。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,  404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,\n",
       "        174,    1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,\n",
       "          4,  892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过整形代号转换到相应的单词来重构原始的文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss although as you will have seen the failed to materialise still the people in a number of countries suffered a series of natural disasters that truly were dreadful eeee'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(tokens_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将它与数据集中原始的文本作对比，除了标点符号和一些单词，如 \"dreaded millennium bug\"之外，几乎是一样的。这是因为我们的词汇表只包含了数据集中10000个频率最多的词，而这3个词出现的频率很低，没有被包括到词汇表中，所以它们被忽略了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"ssss Although, as you will have seen, the dreaded 'millennium bug' failed to materialise, still the people in a number of countries suffered a series of natural disasters that truly were dreadful. eeee\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dest[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练数据\n",
    "\n",
    "现在数据集已经被转换成整形代号（integer-tokens）的序列，它经过填充和截断处理并保存为numpy数组，我们可以用它来训练神经网络。\n",
    "\n",
    "编码器输入的仅仅是由分词器生成的被填充和截断的整形代号序列的numpy数组："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input_data = tokens_src"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器的输入和输出数据是相同的，除了时间偏移了一步。我们可以用相同的数组来节省内存，仅仅需要对内存中相同的数据的采用不同的获取方式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1968800, 54)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_input_data = tokens_dest[:, :-1]\n",
    "decoder_input_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1968800, 54)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_output_data = tokens_dest[:, 1:]\n",
    "decoder_output_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "举个例子，下面的代号序列是相同的，除了它们被移动了一个时间步长。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,  404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,\n",
       "        174,    1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,\n",
       "          4,  892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_input_data[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 404,   19,   43,   26,   20,  618,    1, 1451,    5, 9785,  174,\n",
       "          1,   81,    7,    9,  214,    4,   67, 2200,    9, 1596,    4,\n",
       "        892, 1762,    8, 1480,  107, 5494,    3,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_output_data[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们用分词器将这些序列转换回文本，我们可以看到它们是相同的，除了一个作为文本开始的标记'ssss'。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ssss although as you will have seen the failed to materialise still the people in a number of countries suffered a series of natural disasters that truly were dreadful eeee'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(decoder_input_data[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'tokenizer_dest' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-5649f55db556>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mtokenizer_dest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtokens_to_string\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdecoder_output_data\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0midx\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'tokenizer_dest' is not defined"
     ]
    }
   ],
   "source": [
    "tokenizer_dest.tokens_to_string(decoder_output_data[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建神经网络\n",
    "\n",
    "### 创建编码器\n",
    "\n",
    "首先我们创建神经网络的编码部分，它将整形代号序列映射到“思维向量”（thought vector）。我们将用Keras的函数式（functional）来API完成，我们首先创建神经网络所有层的对象，然后将它们连接，这比Keras中序列式（sequential）API更灵活，当我们尝试更复杂的架构或将连接编码器和解码器的方法时，这个方法很有用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是编码器的输入，它接受按批次的整形代号序列。`None`表示序列的长度可以是任意的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input = Input(shape=(None, ), name='encoder_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是embedding层的输出向量的长度，embedding层将整形代号（integer-tokens ）映射成-1到1的向量的值。所以有相同含义的词映射的向量是相似的。更详细的解释见教程 #20。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是embedding层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_embedding = Embedding(input_dim=num_words,\n",
    "                              output_dim=embedding_size,\n",
    "                              name='encoder_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是GRU内部状态的大小。编码和解码都用了这个大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "state_size = 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建3个GRU层，用于将嵌入向量（embedding-vectors）映射到一个单独的“思维向量” （thought vector），这个向量总结了输入文本的内容。注意最后一个GRU层没有返回序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_gru1 = GRU(state_size, name='encoder_gru1',\n",
    "                   return_sequences=True)\n",
    "encoder_gru2 = GRU(state_size, name='encoder_gru2',\n",
    "                   return_sequences=True)\n",
    "encoder_gru3 = GRU(state_size, name='encoder_gru3',\n",
    "                   return_sequences=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用于连接编码所有层的辅助函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def connect_encoder():\n",
    "    # Start the neural network with its input-layer.\n",
    "    net = encoder_input\n",
    "    \n",
    "    # Connect the embedding-layer.\n",
    "    net = encoder_embedding(net)\n",
    "\n",
    "    # Connect all the GRU-layers.\n",
    "    net = encoder_gru1(net)\n",
    "    net = encoder_gru2(net)\n",
    "    net = encoder_gru3(net)\n",
    "\n",
    "    # This is the output of the encoder.\n",
    "    encoder_output = net\n",
    "    \n",
    "    return encoder_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意编码器是如何使用最后一个GRU层的正常输出作为“思维向量”（thought vector）。研究论文经常使用编码器的最后一个循环层的内部状态作为“思维向量”（thought vector）。但这让实现变得更复杂了并且当使用GRU时也是不需要的。但是如果你使用的是LSTM，那么使用LSTM的内部状态作为“思维向量”是必须的，因为它有两个内部向量，我们需要初始化编码器的LSTM单元的两个内部状态。\n",
    "\n",
    "我们现在可以用这个函数完成编码器层的连接，它在将来会被连接到解码器上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1456: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "encoder_output = connect_encoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建解码器\n",
    "\n",
    "创建解码部分，它将“思维向量”映射到整形代号序列。\n",
    "\n",
    "解码器有两个输入。首先它需要有编码器产生的总结了输入文本内容的“思维向量”（thought vector）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_initial_state = Input(shape=(state_size,),\n",
    "                              name='decoder_initial_state')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器也需要一个整形代号序列作为输入。在训练时，我们将提供完整的整形代号序列，例如对于文本\"ssss once upon a time eeee\"。\n",
    "\n",
    "当我们在翻译新的输入文本时，我们首先喂入序列中\"ssss\"的整形代号（它是一个文本开始的标记），然后与编码器的“思维向量”结合在一起，解码器将有望产生正确的下一个单词，例如\"once\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_input = Input(shape=(None, ), name='decoder_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是embedding层，它将整形代号转换成-1到1的实值向量。注意我们对编码器和解码器有不同的embedding层，因为我们有两个不同词汇表和两个不同分词器对应源语言和目标语言。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_embedding = Embedding(input_dim=num_words,\n",
    "                              output_dim=embedding_size,\n",
    "                              name='decoder_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建解码器的3个GRU。注意它们都返回序列，因为我们最后想要输出一个整形代号的序列，它可以被转换成文本序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_gru1 = GRU(state_size, name='decoder_gru1',\n",
    "                   return_sequences=True)\n",
    "decoder_gru2 = GRU(state_size, name='decoder_gru2',\n",
    "                   return_sequences=True)\n",
    "decoder_gru3 = GRU(state_size, name='decoder_gru3',\n",
    "                   return_sequences=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GRU层最后输出张量的形状是`[batch_size, sequence_length, state_size]`，每一个\"词\"被编码成长度为`state_size`的向量。我们需要将这些转换到可以用我们的词汇表解释的整形代号序列，\n",
    "\n",
    "一种方法是将GRU的输出转换成独热编码的数组。它能起效，但是相当浪费，因为词汇表有10000个词，我们需要有10000个元素的向量，所以我们可以选择最高元素的索引作为整形代号（integer-token）。\n",
    "\n",
    "注意激活函数被设置为`linear`来代替`softmax`，就像我们通常使用于独热编码的输出一样，因为Keras中有似乎有一个bug，所以我们需要自己定制一个损失函数，下面会详细介绍。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_dense = Dense(num_words,\n",
    "                      activation='linear',\n",
    "                      name='decoder_output')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器采用Keras的函数式（functional）API搭建，它允许层的连接更灵活，例如嫁接不同输入到解码器中。这是非常有用的，因为我们必须直接将解码器和编码器相连，但是我们也会连接解码器和另一输入，这样我们可以独立运行它。\n",
    "\n",
    "\n",
    "这个函数将解码器的所有层连接到GRU层的初始状态值的输入。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def connect_decoder(initial_state):\n",
    "    # Start the decoder-network with its input-layer.\n",
    "    net = decoder_input\n",
    "\n",
    "    # Connect the embedding-layer.\n",
    "    net = decoder_embedding(net)\n",
    "    \n",
    "    # Connect all the GRU-layers.\n",
    "    net = decoder_gru1(net, initial_state=initial_state)\n",
    "    net = decoder_gru2(net, initial_state=initial_state)\n",
    "    net = decoder_gru3(net, initial_state=initial_state)\n",
    "\n",
    "    # Connect the final dense layer that converts to\n",
    "    # one-hot encoded arrays.\n",
    "    decoder_output = decoder_dense(net)\n",
    "    \n",
    "    return decoder_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 连接和创建模型\n",
    "\n",
    "我们可以用不同的方式连接编码器和解码器。\n",
    "\n",
    "首先我们将编码器和解码器直接连接，所以它们成了一整个模型，可以用端到端的方式训练。这意味着解码器GRU单元的初始状态被设置成编码器是输出。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_output = connect_decoder(initial_state=encoder_output)\n",
    "\n",
    "model_train = Model(inputs=[encoder_input, decoder_input],\n",
    "                    outputs=[decoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们创建给编码器单独建立一个模型。主要用于将整形代号序列映射到总结了文本内容的“思维向量”。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_encoder = Model(inputs=[encoder_input],\n",
    "                      outputs=[encoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们创建一个单独解码器。它允许我们直接输入解码器的GRU单元初始状态。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_output = connect_decoder(initial_state=decoder_initial_state)\n",
    "\n",
    "model_decoder = Model(inputs=[decoder_input, decoder_initial_state],\n",
    "                      outputs=[decoder_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意到这些模型的编码器和解码器用了相同的权重和变量。我们仅仅改变它们的连接方式。所以一旦完整的模型被训练好了，我们可以用训练好的权重分别运行编码器和解码器。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 损失函数\n",
    "\n",
    "解码器的输出是独热编码的数组序列。为了训练解码器，我们需要提供我们希望看到解码器输出的独热编码数组，然后使用像交叉熵这样的损失函数去训练解码器产生期望的输出。\n",
    "\n",
    "然而，我们的数据集包含整形代号，而不是独热编码数组。每一个独热编码数组有10000个元素，所以将全部的数据集转换成独热数组是相当浪费的。\n",
    "\n",
    "一个好解决方法是使用稀疏的交叉熵（sparse cross-entropy）损失函数，它的内部完成了从整数到独热数组的转换。不幸的是，当和循环神经网络一起使用的时候，Keras似乎存在一个bug，所以下面的方式不起效：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_train.compile(optimizer=optimizer,\n",
    "#                     loss='sparse_categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器输出3阶形状 `[batch_size, sequence_length, num_words]`是张量，它包含了一批长度为`num_words`的独热数组的序列。我们将它与包含了整形序列的2阶`[batch_size, sequence_length]`张量作比较。\n",
    "\n",
    "比较由Tensorflow中的稀疏交叉熵（sparse-cross-entropy）直接完成。下面还有一些需要注意的事项。\n",
    "\n",
    "首先，为了提高数值的稳定性，损失函数在内部计算softamax，这也是为什么我们在解码器网络最后的全连接层中用了线性激活函数。\n",
    "\n",
    "第二，TensoFlow的损失函数输出的是形状为`[batch_size, sequence_length]`的二阶张量。但是它最终要被转换为一个单独标量值，它的梯度可以被TensorFlow推导并可以用梯度下降来优化。Keras支持在批处理中对损失值进行加权，但描述的不清楚，所以为了确保我们计算的损失函数经过了整个批处理和整个序列，我们手工计算损失平均值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sparse_cross_entropy(y_true, y_pred):\n",
    "    \"\"\"\n",
    "    Calculate the cross-entropy loss between y_true and y_pred.\n",
    "    \n",
    "    y_true is a 2-rank tensor with the desired output.\n",
    "    The shape is [batch_size, sequence_length] and it\n",
    "    contains sequences of integer-tokens.\n",
    "\n",
    "    y_pred is the decoder's output which is a 3-rank tensor\n",
    "    with shape [batch_size, sequence_length, num_words]\n",
    "    so that for each sequence in the batch there is a one-hot\n",
    "    encoded array of length num_words.\n",
    "    \"\"\"\n",
    "\n",
    "    # Calculate the loss. This outputs a\n",
    "    # 2-rank tensor of shape [batch_size, sequence_length]\n",
    "    loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_true,\n",
    "                                                          logits=y_pred)\n",
    "\n",
    "    # Keras may reduce this across the first axis (the batch)\n",
    "    # but the semantics are unclear, so to be sure we use\n",
    "    # the loss across the entire 2-rank tensor, we reduce it\n",
    "    # to a single scalar with the mean function.\n",
    "    loss_mean = tf.reduce_mean(loss)\n",
    "\n",
    "    return loss_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 编译训练模型\n",
    "\n",
    "我们在之前的教程中用了Adam优化器，但是在一些试验中发现它与循环神经网络有冲突。RMSprop在这似乎表现的更好一些。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "optimizer = RMSprop(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras在这里似乎又有一个bug，它不能自动的推断正确的编码器输出的形状。因此我们需要手动创建一个占位符变量来作为解码器的输出。它的形状被设置成`(None, None)`，意味着这批可以有任意数量的序列，任意数量的整形代号。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_target = tf.placeholder(dtype='int32', shape=(None, None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在可以用自己定制的损失函数编译模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1557: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "model_train.compile(optimizer=optimizer,\n",
    "                    loss=sparse_cross_entropy,\n",
    "                    target_tensors=[decoder_target])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 回调函数\n",
    "在训练时，我们希望保存checkpoints并将进展记录到TensorBoard中，所以我们用了Keras中的相应回调。\n",
    "\n",
    "下面是在训练时保存checkpoints的回调。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_checkpoint = '21_checkpoint.keras'\n",
    "callback_checkpoint = ModelCheckpoint(filepath=path_checkpoint,\n",
    "                                      monitor='val_loss',\n",
    "                                      verbose=1,\n",
    "                                      save_weights_only=True,\n",
    "                                      save_best_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是用于在验证集上模型表现的更糟时停止优化的回调。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "callback_early_stopping = EarlyStopping(monitor='val_loss',\n",
    "                                        patience=3, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是在训练时写TensorBoard日志的回调。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "callback_tensorboard = TensorBoard(log_dir='./21_logs/',\n",
    "                                   histogram_freq=0,\n",
    "                                   write_graph=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [callback_early_stopping,\n",
    "             callback_checkpoint,\n",
    "             callback_tensorboard]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载 Checkpoint\n",
    "\n",
    "我们可以重新加载最后保存的checkpoint ，所以当我们每当我们想要使用这个模型时，不需要再训练它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    model_train.load_weights(path_checkpoint)\n",
    "except Exception as error:\n",
    "    print(\"Error trying to load checkpoint.\")\n",
    "    print(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练模型\n",
    "\n",
    "我们将数据封装在命名的字典中，所以我们确认数据被正确地分配给模型的输入和输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_data = \\\n",
    "{\n",
    "    'encoder_input': encoder_input_data,\n",
    "    'decoder_input': decoder_input_data\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_data = \\\n",
    "{\n",
    "    'decoder_output': decoder_output_data\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们想要10000序列的验证集，但是Keras需要这个数字是一个分数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0050792360828931325"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_split = 10000 / len(encoder_input_data)\n",
    "validation_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们可以训练这个模型。在GTX 1070 GPU上训练一个epoch大约花费1小时。在训练时你可能需要运行10个epoch或者更多。在10个epochs后，在训练集上损失大约是1.10，在验证集上的损失大约是1.15。\n",
    "\n",
    "注意这个被选为640（512+128）的批大小，当GPU内存现在在8GB时，它能让GPU接近100%的运行。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_train.fit(x=x_data,\n",
    "                y=y_data,\n",
    "                batch_size=640,\n",
    "                epochs=10,\n",
    "                validation_split=validation_split,\n",
    "                callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 翻译文本\n",
    "\n",
    "这个函数将一个文本从源语言转换到目标语言并且可选地打印正确的翻译。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate(input_text, true_output_text=None):\n",
    "    \"\"\"Translate a single text-string.\"\"\"\n",
    "\n",
    "    # Convert the input-text to integer-tokens.\n",
    "    # Note the sequence of tokens has to be reversed.\n",
    "    # Padding is probably not necessary.\n",
    "    input_tokens = tokenizer_src.text_to_tokens(text=input_text,\n",
    "                                                reverse=True,\n",
    "                                                padding=True)\n",
    "    \n",
    "    # Get the output of the encoder's GRU which will be\n",
    "    # used as the initial state in the decoder's GRU.\n",
    "    # This could also have been the encoder's final state\n",
    "    # but that is really only necessary if the encoder\n",
    "    # and decoder use the LSTM instead of GRU because\n",
    "    # the LSTM has two internal states.\n",
    "    initial_state = model_encoder.predict(input_tokens)\n",
    "\n",
    "    # Max number of tokens / words in the output sequence.\n",
    "    max_tokens = tokenizer_dest.max_tokens\n",
    "\n",
    "    # Pre-allocate the 2-dim array used as input to the decoder.\n",
    "    # This holds just a single sequence of integer-tokens,\n",
    "    # but the decoder-model expects a batch of sequences.\n",
    "    shape = (1, max_tokens)\n",
    "    decoder_input_data = np.zeros(shape=shape, dtype=np.int)\n",
    "\n",
    "    # The first input-token is the special start-token for 'ssss '.\n",
    "    token_int = token_start\n",
    "\n",
    "    # Initialize an empty output-text.\n",
    "    output_text = ''\n",
    "\n",
    "    # Initialize the number of tokens we have processed.\n",
    "    count_tokens = 0\n",
    "\n",
    "    # While we haven't sampled the special end-token for ' eeee'\n",
    "    # and we haven't processed the max number of tokens.\n",
    "    while token_int != token_end and count_tokens < max_tokens:\n",
    "        # Update the input-sequence to the decoder\n",
    "        # with the last token that was sampled.\n",
    "        # In the first iteration this will set the\n",
    "        # first element to the start-token.\n",
    "        decoder_input_data[0, count_tokens] = token_int\n",
    "\n",
    "        # Wrap the input-data in a dict for clarity and safety,\n",
    "        # so we are sure we input the data in the right order.\n",
    "        x_data = \\\n",
    "        {\n",
    "            'decoder_initial_state': initial_state,\n",
    "            'decoder_input': decoder_input_data\n",
    "        }\n",
    "\n",
    "        # Note that we input the entire sequence of tokens\n",
    "        # to the decoder. This wastes a lot of computation\n",
    "        # because we are only interested in the last input\n",
    "        # and output. We could modify the code to return\n",
    "        # the GRU-states when calling predict() and then\n",
    "        # feeding these GRU-states as well the next time\n",
    "        # we call predict(), but it would make the code\n",
    "        # much more complicated.\n",
    "\n",
    "        # Input this data to the decoder and get the predicted output.\n",
    "        decoder_output = model_decoder.predict(x_data)\n",
    "\n",
    "        # Get the last predicted token as a one-hot encoded array.\n",
    "        token_onehot = decoder_output[0, count_tokens, :]\n",
    "        \n",
    "        # Convert to an integer-token.\n",
    "        token_int = np.argmax(token_onehot)\n",
    "\n",
    "        # Lookup the word corresponding to this integer-token.\n",
    "        sampled_word = tokenizer_dest.token_to_word(token_int)\n",
    "\n",
    "        # Append the word to the output-text.\n",
    "        output_text += \" \" + sampled_word\n",
    "\n",
    "        # Increment the token-counter.\n",
    "        count_tokens += 1\n",
    "\n",
    "    # Sequence of tokens output by the decoder.\n",
    "    output_tokens = decoder_input_data[0]\n",
    "    \n",
    "    # Print the input-text.\n",
    "    print(\"Input text:\")\n",
    "    print(input_text)\n",
    "    print()\n",
    "\n",
    "    # Print the translated output-text.\n",
    "    print(\"Translated text:\")\n",
    "    print(output_text)\n",
    "    print()\n",
    "\n",
    "    # Optionally print the true translated text.\n",
    "    if true_output_text is not None:\n",
    "        print(\"True output text:\")\n",
    "        print(true_output_text)\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 举例\n",
    "\n",
    "从训练数据中翻译一个文本。翻译的相当不错。它与训练数据中翻译不一样，但是实际内容是相似的。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.\n",
      "\n",
      "Translated text:\n",
      " you have expressed a wish for a debate on this matter during the part session eeee\n",
      "\n",
      "True output text:\n",
      "ssss You have requested a debate on this subject in the course of the next few days, during this part-session. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx],\n",
    "          true_output_text=data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是另一个例子，它也是一个合理的翻译，尽管它错误的翻译了自然灾害。注意\"countries of the European Union\" 被翻译为 \"member states\"，在这个文本中他们是同义词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.\n",
      "\n",
      "Translated text:\n",
      " in the meantime i also asked for a minute's silence on the memory of victims of the atrocities that have been committed in the member states eeee\n",
      "\n",
      "True output text:\n",
      "ssss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 4\n",
    "translate(input_text=data_src[idx],\n",
    "          true_output_text=data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将训练集中两个文本连接起来。模型先将合并的文本输入到编码器并似乎产生了对两个文本总结相当好的“思维向量”，所以解码器可以产生合理的翻译。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.\n",
      "\n",
      "Translated text:\n",
      " you have expressed a wish for a vote on this question during the vote on thursday and in the end i would also like to ask you to pay tribute to the memory of a tragedy in the case of the victims of the various member states eeee\n",
      "\n",
      "True output text:\n",
      "ssss You have requested a debate on this subject in the course of the next few days, during this part-session. eeeessss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx] + data_src[idx+1],\n",
    "          true_output_text=data_dest[idx] + data_dest[idx+1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们反转这两个文本的顺序，对于后一种文本来说，含义并不十分清楚。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "I mellemtiden ønsker jeg - som også en del kolleger har anmodet om - at vi iagttager et minuts stilhed til minde om ofrene for bl.a. stormene i de medlemslande, der blev ramt.De har udtrykt ønske om en debat om dette emne i løbet af mødeperioden.\n",
      "\n",
      "Translated text:\n",
      " in the meantime i would also like to ask you to remember that we have received a silence on the victims of the floods in the member states of the european union which have been particularly sensitive to this debate in the house eeee\n",
      "\n",
      "True output text:\n",
      "ssss In the meantime, I should like to observe a minute' s silence, as a number of Members have requested, on behalf of all the victims concerned, particularly those of the terrible storms, in the various countries of the European Union. eeeessss You have requested a debate on this subject in the course of the next few days, during this part-session. eeee\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idx = 3\n",
    "translate(input_text=data_src[idx+1] + data_src[idx],\n",
    "          true_output_text=data_dest[idx+1] + data_dest[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是我自己编的例子。翻译的结果相当糟糕\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "der var engang et land der hed Danmark\n",
      "\n",
      "Translated text:\n",
      " there was a country that denmark was once again eeee\n",
      "\n",
      "True output text:\n",
      "Once there was a country named Denmark\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"der var engang et land der hed Danmark\",\n",
    "          true_output_text='Once there was a country named Denmark')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我再编一个。尽管它是更复杂的文本，翻译的还好了一些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Idag kan man læse i avisen at Danmark er blevet fornuftigt\n",
      "\n",
      "Translated text:\n",
      " can you read in the newspapers that denmark has been sensible eeee\n",
      "\n",
      "True output text:\n",
      "Today you can read in the newspaper that Denmark has become sensible.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"Idag kan man læse i avisen at Danmark er blevet fornuftigt\",\n",
    "          true_output_text=\"Today you can read in the newspaper that Denmark has become sensible.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是丹麦歌曲中的一个文本。它在丹麦语中没有很多含义。翻译的支离破碎，因为其中的几个词不在词汇表中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Hvem spæner ud af en butik og tygger de stærkeste bolcher?\n",
      "\n",
      "Translated text:\n",
      " who is by a and by the powerful eeee\n",
      "\n",
      "True output text:\n",
      "Who runs out of a shop and chews the strongest bon-bons?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "translate(input_text=\"Hvem spæner ud af en butik og tygger de stærkeste bolcher?\",\n",
    "          true_output_text=\"Who runs out of a shop and chews the strongest bon-bons?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "这份教程展示了用两循环神经网络搭建编码/解码模型去完成人类语言的机器翻译的基本思想。用了欧盟Europarl数据集做了演示。\n",
    "\n",
    "这个模型可以对一些文本产生合理的翻译，但对另一些文本则不适用。更好的网络结构和训练更多epoch可能可以提高性能。现在已知有非常多先进的模型可以提高翻译的质量。\n",
    "\n",
    "然而，注意到这些模型并没有真正理解人类的语言是很重要的。这些模型不知道这些词的实际含义。这些模型仅仅是非常高级的函数近似器，它们可以在整数代号的序列之间进行映射。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习\n",
    "\n",
    "下面使一些可能会让你提升TensorFlow技能的一些建议练习。为了学习如何更合适地使用TensorFlow，实践经验是很重要的。\n",
    "\n",
    "在你对这个Notebook进行修改之前，可能需要先备份一下。\n",
    "\n",
    "* 接着训练10个epoch。翻译是否有所提高？\n",
    "* 增加词汇表的大小。翻译是否有所提高？源语言和目标语言词汇表的大小不同有什么意义吗？\n",
    "* 找到另一个数据集并与Europarl一起使用。\n",
    "* 改变网络结构，例如改变GRU层的状态大小，GRU的层数，embedding层的大小等等。翻译是否有所提高？\n",
    "* 用教程 #19中的超参数优化去自动找到最好的超参数。\n",
    "* 翻译文本时，你能不能把解码器的输出样本看作是概率分布，来代替使用 `np.argmax()`对下一个整形代号的采样？注意解码器的输出不受softmax的限制，所以你必须先把它变成概率分布。\n",
    "* 你可以通过做这个采样来生成多个序列吗？你能找到一种方法来选择这些不同序列中最好的吗？\n",
    "* 禁用源语言中单词的反转。翻译是否有所提高？\n",
    "* 什么是双向Bi-DirectionalGRU，你可以在这里使用它吗？\n",
    "* 我们用编码器的GRU的输出作为解码器GRU的初始状态。研究文献通常使用的是LSTM而不是GRU，所以他们使用了编码器的状态来代替它的输出作为解码器的初始状态。你可以将代码改写成用编码器的状态作为解码器的初始状态吗？有理由这么做吗，或者编码器的输出足以作为解码器的初始状态？\n",
    "* 是否有可能将多个编码器和解码器连接到一个神经网络中，这样你就可以用不同的语言进行训练并允许直接翻译，例如丹麦语到波兰语，德语和法语。\n",
    "* 向朋友解释程序是如何工作的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}