fastai_NLP/imdb.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "# 多行输出\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDB\n",
    "\n",
    "作者只考虑了高度两极化的评论。负面评价得分≤4分(满分10分)，正面评价得分≥7分(满分10分)。中性评审不包括在数据集中。数据集分为训练集和测试集。培训集是相同的25000个带标签的评论"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 获取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/lab/.fastai/data/imdb_sample')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/lab/.fastai/data/imdb_sample/texts.csv')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'texts.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = TextDataBunch.from_csv(path, 'texts.csv', text_cols=1, label_cols=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelList (3 items)\n",
       "x: TextList\n",
       "xxbos ' xxmaj major xxmaj payne ' is a film about a major who makes life a living xxmaj hell for his small group of boys in the marines . xxmaj this film does not really have a lot to offer , but it provides several hilarious moments that are well - worth a watch . xxmaj do n't expect it to be a memorable film , however . xxmaj just expect to laugh your way through the film and at the expense of other people . xxmaj the confrontation between xxmaj major xxmaj payne and the chubby boy were hilarious , and that 's really all i remember about the film except for the boys wanting revenge on xxmaj major xxmaj payne . xxmaj again , it is not a great film , and it is probably best watched on a rainy day when you need some laughter .,xxbos xxmaj prince stars as ' the xxmaj kid ' in this semi - autobiographical film of a talented , but narcissistic young musician who has a less then stellar home life . xxmaj true the acting leaves a tad to be desired ( barring xxmaj morris xxmaj day and especially xxmaj clarence xxmaj williams who are both pitch perfect ) , but the movie is still great and among the best to come out of the 1980s . xxmaj it has the best soundtrack of xxup any movie of the last 50 years at least , highly quotable lines , and the xxunk scene is xxup hilarious ! ! xxmaj plus xxmaj apollonia is just simply xxup stunning . xxmaj on an unrelated not , when i saw xxmaj prince in concert in 2004 he blew down the stadium . xxmaj he is an expert xxunk and it was one of the best xxunk that i 've experienced . \n",
       " \n",
       "  xxmaj my xxmaj grade : a \n",
       " \n",
       "  xxup dvd xxmaj extras : xxmaj disc 1 ) xxmaj commentary with xxmaj director xxmaj albert xxmaj xxunk , xxmaj producer xxmaj robert xxmaj xxunk , & xxmaj director of xxmaj photography xxmaj donald xxmaj xxunk ; xxmaj theatrical xxmaj trailer ; xxmaj trailers for \" xxmaj under the xxmaj cherry xxmaj moon \" and \" xxmaj xxunk xxmaj bridge \" xxmaj disc 2 ) a 12 minute featurette on the xxmaj first xxmaj avenue xxmaj club ; \" xxmaj purple xxmaj rain : xxmaj xxunk xxmaj pass ( a half hour featurette on the movie which i 'll review later on it 's page ) ; \" xxmaj xxunk , xxmaj xxunk , and xxmaj revolution : the xxmaj impact and xxmaj influence of xxmaj purple xxmaj rain \" 10 minute featurette ; 30 minutes of xxup mtv 's xxmaj premiere xxmaj footage ( when xxup mtv did n't suck xxunk balls ) ; 5 xxmaj prince music videos ( xxmaj let 's xxmaj go xxmaj crazy , xxmaj take xxmaj me xxmaj with xxmaj you , xxmaj when xxmaj xxunk xxmaj cry , i xxmaj would xxmaj die 4 u / xxmaj baby i 'm a xxmaj star , and xxmaj purple xxmaj rain ) ; 2 xxmaj videos by xxmaj the xxmaj time ( xxmaj jungle xxmaj love and xxmaj the xxmaj bird ) ; and a music video for \" xxmaj sex xxmaj shooter \" by xxmaj apollonia 6 \n",
       " \n",
       "  xxmaj eye xxmaj candy : xxmaj apollonia shows her fine ass xxunk,xxbos xxmaj just finished watching this movie for maybe the xxunk or 8th time , picked it up one night previously viewed at xxmaj blockbuster and absolutely loved it , i 've shown it to 4 people so far and they have enjoyed it as well . xxmaj avoid of all the xxmaj hollywood glamour , special effects and stress on the \" shock factor \" , this independent film by xxmaj paul xxup f. xxmaj ryan hits the nail on the head in dealing with the after affects of traumatic situations . xxmaj taking place after a high school shooting , two characters xxmaj alicia ( xxmaj busy xxmaj philipps ) and xxmaj deanna ( xxmaj xxunk xxmaj christensen ) form an unlikely bond . xxmaj alicia , the girl with the stone heart , the xxmaj goth who has a xxunk attitude to life xxunk xxmaj deanna to overcome the issues of life and death and living in the aftermath . xxmaj meanwhile xxmaj deanna attempts to help xxmaj alicia to see some of the xxunk and light in the world again . xxmaj not xxunk on the shocking event of the shooting , but on the xxunk relationships amongst those who survived it sets this movie apart . xxmaj despite its low - budget and short filming time , this movie is far from cheesy . xxmaj ryan pays respect to a situation he has never xxunk and attempts to xxunk into the human psyche . xxmaj with an amazing up and coming actress , xxmaj philipps , adds the necessary xxunk to the dialogue and overall feel for the film and xxmaj christensen helps to balance out the \" doom and gloom \" feeling this movie may have . xxmaj overall , i recommend this movie and if you enjoy the topic of school xxunk and violence and learning more about it , i also suggest the documentary \" xxmaj it 's a xxmaj girls xxmaj world \" put out by xxup xxunk in 2004 , which deals with the topic of social bullying , comparing and xxunk two groups of girls one in xxmaj xxunk , xxmaj xxunk and the other in xxmaj victoria , xxmaj british xxmaj columbia , the group of friends and acquaintances of xxmaj dawn xxmaj marie xxmaj xxunk a 14 year old girl who killed herself after being brutally xxunk .\n",
       "y: CategoryList\n",
       "negative,positive,positive\n",
       "Path: /home/lab/.fastai/data/imdb_sample"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 标记化 token\n",
    "- 分词，标点符号，有特殊含义的标记"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- fastai 的特殊标记\n",
    "\n",
    "```\n",
    "defaults.text_spec_tok = [UNK,PAD,BOS,FLD,TK_MAJ,TK_UP,TK_REP,TK_WREP]\n",
    "\n",
    "The rules are all listed below, here is the meaning of the special tokens:\n",
    "\n",
    "UNK (xxunk) is for an unknown word (one that isn't present in the current vocabulary)\n",
    "PAD (xxpad) is the token used for padding, if we need to regroup several texts of different lengths in a batch\n",
    "BOS (xxbos) represents the beginning of a text in your dataset\n",
    "FLD (xxfld) is used if you set mark_fields=True in your TokenizeProcessor to separate the different fields of texts (if your texts are loaded from several columns in a dataframe)\n",
    "TK_MAJ (xxmaj) is used to indicate the next word begins with a capital in the original text\n",
    "TK_UP (xxup) is used to indicate the next word is written in all caps in the original text\n",
    "TK_REP (xxrep) is used to indicate the next character is repeated n times in the original text (usage xxrep n {char})\n",
    "TK_WREP(xxwrep) is used to indicate the next word is repeated n times in the original text (usage xxwrep n {word})\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "799"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "201"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.train_ds)\n",
    "len(data.valid_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8861, 19162)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos), len(data.vocab.stoi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "stoi (string-to-int) is larger than itos (int-to-string).This is because many words are **mapping to unknown**. We can confirm here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxunk'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.itos[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 0 是未知词\n",
    "unk = []\n",
    "for word, num in data.vocab.stoi.items():\n",
    "    if num==0:\n",
    "        unk.append(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10302"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(unk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10301"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "19162 - 8861"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'dumpster',\n",
       " 'showman',\n",
       " 'concerts',\n",
       " 'magnoli',\n",
       " 'cavallo',\n",
       " 'thorin',\n",
       " 'grafitti',\n",
       " 'bachstage',\n",
       " 'riffs']"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unk[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6063"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.stoi['quotable']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'quotable'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.itos[6063]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.stoi['xdfs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxunk'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.itos[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "term-document 矩阵将文档表示为一个“bag of words”，也就是说，我们不记录单词的顺序，只记录单词出现的顺序(以及出现的频率)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 稀疏矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 稀疏矩阵的类型\n",
    "    - coordinate-wise (scipy calls COO)\n",
    "    - compressed sparse row (CSR)\n",
    "    - compressed sparse column (CSC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- n-gram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "veczr = CountVectorizer(ngram_range=(1,3), preprocessor=noop, tokenizer=noop, max_features=800000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextList (2 items)\n",
       "xxbos ' xxmaj major xxmaj payne ' is a film about a major who makes life a living xxmaj hell for his small group of boys in the marines . xxmaj this film does not really have a lot to offer , but it provides several hilarious moments that are well - worth a watch . xxmaj do n't expect it to be a memorable film , however . xxmaj just expect to laugh your way through the film and at the expense of other people . xxmaj the confrontation between xxmaj major xxmaj payne and the chubby boy were hilarious , and that 's really all i remember about the film except for the boys wanting revenge on xxmaj major xxmaj payne . xxmaj again , it is not a great film , and it is probably best watched on a rainy day when you need some laughter .,xxbos xxmaj prince stars as ' the xxmaj kid ' in this semi - autobiographical film of a talented , but narcissistic young musician who has a less then stellar home life . xxmaj true the acting leaves a tad to be desired ( barring xxmaj morris xxmaj day and especially xxmaj clarence xxmaj williams who are both pitch perfect ) , but the movie is still great and among the best to come out of the 1980s . xxmaj it has the best soundtrack of xxup any movie of the last 50 years at least , highly quotable lines , and the xxunk scene is xxup hilarious ! ! xxmaj plus xxmaj apollonia is just simply xxup stunning . xxmaj on an unrelated not , when i saw xxmaj prince in concert in 2004 he blew down the stadium . xxmaj he is an expert xxunk and it was one of the best xxunk that i 've experienced . \n",
       " \n",
       "  xxmaj my xxmaj grade : a \n",
       " \n",
       "  xxup dvd xxmaj extras : xxmaj disc 1 ) xxmaj commentary with xxmaj director xxmaj albert xxmaj xxunk , xxmaj producer xxmaj robert xxmaj xxunk , & xxmaj director of xxmaj photography xxmaj donald xxmaj xxunk ; xxmaj theatrical xxmaj trailer ; xxmaj trailers for \" xxmaj under the xxmaj cherry xxmaj moon \" and \" xxmaj xxunk xxmaj bridge \" xxmaj disc 2 ) a 12 minute featurette on the xxmaj first xxmaj avenue xxmaj club ; \" xxmaj purple xxmaj rain : xxmaj xxunk xxmaj pass ( a half hour featurette on the movie which i 'll review later on it 's page ) ; \" xxmaj xxunk , xxmaj xxunk , and xxmaj revolution : the xxmaj impact and xxmaj influence of xxmaj purple xxmaj rain \" 10 minute featurette ; 30 minutes of xxup mtv 's xxmaj premiere xxmaj footage ( when xxup mtv did n't suck xxunk balls ) ; 5 xxmaj prince music videos ( xxmaj let 's xxmaj go xxmaj crazy , xxmaj take xxmaj me xxmaj with xxmaj you , xxmaj when xxmaj xxunk xxmaj cry , i xxmaj would xxmaj die 4 u / xxmaj baby i 'm a xxmaj star , and xxmaj purple xxmaj rain ) ; 2 xxmaj videos by xxmaj the xxmaj time ( xxmaj jungle xxmaj love and xxmaj the xxmaj bird ) ; and a music video for \" xxmaj sex xxmaj shooter \" by xxmaj apollonia 6 \n",
       " \n",
       "  xxmaj eye xxmaj candy : xxmaj apollonia shows her fine ass xxunk\n",
       "Path: /home/lab/.fastai/data/imdb_sample"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = data.train_dl.x\n",
    "docs[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,   64,    5,  546, ...,  342,   62, 1570,   11])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs.vocab.itos[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_words = [[docs.vocab.itos[o] for o in doc.data] for doc in data.train_dl.x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_words = [[docs.vocab.itos[o] for o in doc.data] for doc in data.valid_dl.x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 统计词频"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ngram_doc = veczr.fit_transform(train_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [3, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [2, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       ...,\n",
       "       [2, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [5, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [5, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [4, 0, 0, 1, ..., 0, 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ngram_doc.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('£ 200 million', 283503),\n",
       " ('£ 200', 283502),\n",
       " ('£ 1 in', 283501),\n",
       " ('£ 1', 283500),\n",
       " ('£', 283499),\n",
       " ('\\x96 xxmaj xxunk', 283498),\n",
       " ('\\x96 xxmaj trained', 283497),\n",
       " ('\\x96 xxmaj setting', 283496),\n",
       " ('\\x96 xxmaj robert', 283495),\n",
       " ('\\x96 xxmaj rhys', 283494)]"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(veczr.vocabulary_.items(), key=lambda x: x[1], reverse=True)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_ngram_doc = veczr.transform(valid_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [3, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [4, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       ...,\n",
       "       [2, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [2, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, ..., 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_ngram_doc.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 使用词频进行分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = veczr.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['so xxunk at',\n",
       " 'so xxunk by',\n",
       " 'so xxunk cheesy',\n",
       " 'so xxunk claimed',\n",
       " 'so xxunk disappointed']"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[200000:200005]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = data.train_ds.y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CategoryList (799 items)\n",
       "negative,positive,positive,negative,positive\n",
       "Path: /home/lab/.fastai/data/imdb_sample"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 0, ..., 0, 0, 1, 0])"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y\n",
    "y.items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mCall signature:\u001b[0m  \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mType:\u001b[0m            ufunc\n",
       "\u001b[0;31mString form:\u001b[0m     <ufunc 'sign'>\n",
       "\u001b[0;31mFile:\u001b[0m            ~/Softwares/miniconda3/envs/fastai/lib/python3.6/site-packages/numpy/__init__.py\n",
       "\u001b[0;31mDocstring:\u001b[0m      \n",
       "sign(x, /, out=None, *, where=True, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj])\n",
       "\n",
       "Returns an element-wise indication of the sign of a number.\n",
       "\n",
       "The `sign` function returns ``-1 if x < 0, 0 if x==0, 1 if x > 0``.  nan\n",
       "is returned for nan inputs.\n",
       "\n",
       "For complex inputs, the `sign` function returns\n",
       "``sign(x.real) + 0j if x.real != 0 else sign(x.imag) + 0j``.\n",
       "\n",
       "complex(nan, 0) is returned for complex nan inputs.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "x : array_like\n",
       "    Input values.\n",
       "out : ndarray, None, or tuple of ndarray and None, optional\n",
       "    A location into which the result is stored. If provided, it must have\n",
       "    a shape that the inputs broadcast to. If not provided or `None`,\n",
       "    a freshly-allocated array is returned. A tuple (possible only as a\n",
       "    keyword argument) must have length equal to the number of outputs.\n",
       "where : array_like, optional\n",
       "    Values of True indicate to calculate the ufunc at that position, values\n",
       "    of False indicate to leave the value in the output alone.\n",
       "**kwargs\n",
       "    For other keyword-only arguments, see the\n",
       "    :ref:`ufunc docs <ufuncs.kwargs>`.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "y : ndarray\n",
       "    The sign of `x`.\n",
       "    This is a scalar if `x` is a scalar.\n",
       "\n",
       "Notes\n",
       "-----\n",
       "There is more than one definition of sign in common use for complex\n",
       "numbers.  The definition used here is equivalent to :math:`x/\\sqrt{x*x}`\n",
       "which is different from a common alternative, :math:`x/|x|`.\n",
       "\n",
       "Examples\n",
       "--------\n",
       ">>> np.sign([-5., 4.5])\n",
       "array([-1.,  1.])\n",
       ">>> np.sign(0)\n",
       "0\n",
       ">>> np.sign(5-2j)\n",
       "(1+0j)\n",
       "\u001b[0;31mClass docstring:\u001b[0m\n",
       "Functions that operate element by element on whole arrays.\n",
       "\n",
       "To see the documentation for a specific ufunc, use `info`.  For\n",
       "example, ``np.info(np.sin)``.  Because ufuncs are written in C\n",
       "(for speed) and linked into Python with NumPy's ufunc facility,\n",
       "Python's help() function finds this page whenever help() is called\n",
       "on a ufunc.\n",
       "\n",
       "A detailed explanation of ufuncs can be found in the docs for :ref:`ufuncs`.\n",
       "\n",
       "Calling ufuncs:\n",
       "===============\n",
       "\n",
       "op(*x[, out], where=True, **kwargs)\n",
       "Apply `op` to the arguments `*x` elementwise, broadcasting the arguments.\n",
       "\n",
       "The broadcasting rules are:\n",
       "\n",
       "* Dimensions of length 1 may be prepended to either array.\n",
       "* Arrays may be repeated along dimensions of length 1.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "*x : array_like\n",
       "    Input arrays.\n",
       "out : ndarray, None, or tuple of ndarray and None, optional\n",
       "    Alternate array object(s) in which to put the result; if provided, it\n",
       "    must have a shape that the inputs broadcast to. A tuple of arrays\n",
       "    (possible only as a keyword argument) must have length equal to the\n",
       "    number of outputs; use `None` for uninitialized outputs to be\n",
       "    allocated by the ufunc.\n",
       "where : array_like, optional\n",
       "    Values of True indicate to calculate the ufunc at that position, values\n",
       "    of False indicate to leave the value in the output alone.  Note that if\n",
       "    an uninitialized return array is created via the default ``out=None``,\n",
       "    then the elements where the values are False will remain uninitialized.\n",
       "**kwargs\n",
       "    For other keyword-only arguments, see the :ref:`ufunc docs <ufuncs.kwargs>`.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "r : ndarray or tuple of ndarray\n",
       "    `r` will have the shape that the arrays in `x` broadcast to; if `out` is\n",
       "    provided, it will be returned. If not, `r` will be allocated and\n",
       "    may contain uninitialized values. If the function has more than one\n",
       "    output, then the result will be a tuple of arrays.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?np.sign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(799, 283504)"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ngram_doc.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 是否出现某词比较重要，和出现的频率没有太大的关系"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       ...,\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 0, ..., 0, 0, 0, 0],\n",
       "       [1, 0, 0, 1, ..., 0, 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ngram_doc.sign().toarray()  # one-hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0, 1}"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set.union(*[set(c) for c in train_ngram_doc.sign().toarray()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lab/Softwares/miniconda3/envs/fastai/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "model = LogisticRegression(C=0.1, dual=True)\n",
    "# 使用词频数据的符号函数\n",
    "model.fit(train_ngram_doc.sign(), y.items);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, ..., 1, 0, 0, 1])"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model.predict(val_ngram_doc.sign())\n",
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7910447761194029"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(preds.T == data.valid_dl.y.items).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 不使用二值化或者 one-hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lab/Softwares/miniconda3/envs/fastai/lib/python3.6/site-packages/sklearn/svm/base.py:931: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "model = LogisticRegression(C=0.1, dual=True)\n",
    "# 使用词频数据的符号函数\n",
    "model.fit(train_ngram_doc, y.items);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, ..., 1, 0, 0, 0])"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model.predict(val_ngram_doc)\n",
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7562189054726368"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(preds.T == data.valid_dl.y.items).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 说明词频对语义分类是一个几乎无效的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}