BookThiefSentimentAnalysis.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring The Book Thief(Movie) Sentiment Analysis using Bilinear Logistic Regression\n",
    "In this notebook the following shall be done:\n",
    "* Obtain labelled training data from Kaggle\n",
    "* Define a cleaning function for text cleaning\n",
    "* Choose a model(Bilinear Logistic Regression), split the data into test and train data\n",
    "* Build and train the model\n",
    "* Scrape the BookThief Reviews from IMDB\n",
    "* Predict the polarity of the reviews"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis\n",
    "Sentiment analysis is an NLP process of identifying the tone expressed in a text. Is it positive,negative or neutral. \n",
    "Opinion mining is especially useful for feedback on how customers feel about a brand or product and helps a business understand better customer needs."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP: Natural Language Processing\n",
    "This is a branch of machine learning that allows a computer to read, understand and draw meaning from human understandable text. We understand words and computers understand text. Human readable text has to undergo a couple of transformations before machines can make sense of it. These include:\n",
    "* Segmentation - This is the breaking down of a document into its constituent sentences\n",
    "* Tokenization - Breaking down of sentences into words\n",
    "* Removal of Stopwords - Stopwords are a list of words to be filtered out from the document as they are insignificant due to their popularity when it comes to natural language processing.They include prepositions, pronouns, conjunctions etc.\n",
    "*Lemmatization - This is the process of grouping together inflected forms of a word so they can be analysed as a single word. Example: Is, Are and Am all come under the verb 'to be' but in different persons."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Logistic Regression to get the tone\n",
    "There are a variety of models that can be used to get the tone in a text. Some include SVMs, Random Forests and Naive Bayes. For this project, we will be identifying one of two tones, negative or positive.\n",
    "\n",
    "Logistic regression predicts the probability of an event or outcome. The possible outcomes being categorical -meaning they are stored and identified based on names or labels given to them. In our case, we have prelabelled data where positive = 1 and negative = 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Data columns (total 3 columns):\n",
      " #   Column     Non-Null Count  Dtype \n",
      "---  ------     --------------  ----- \n",
      " 0   id         25000 non-null  object\n",
      " 1   sentiment  25000 non-null  int64 \n",
      " 2   review     25000 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 586.1+ KB\n"
     ]
    }
   ],
   "source": [
    "#importing the pandas library, reading and exploring the dataset\n",
    "import pandas as pd\n",
    "dataset = pd.read_csv(\"labeledTrainData.tsv\", sep='\\t', index_col=None)\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment                                             review\n",
       "0  5814_8          1  With all this stuff going down at the moment w...\n",
       "1  2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "2  7759_3          0  The film starts with a manager (Nicholas Bell)...\n",
       "3  3630_4          0  It must be assumed that those who praised this...\n",
       "4  9495_8          1  Superbly trashy and wondrously unpretentious 8..."
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#exploring the dataset \n",
    "dataset.head()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#exploring the number of sentiments by count\n",
    "dataset['sentiment'].nunique()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    12500\n",
       "0    12500\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#exploring the distribution number of sentiments by count\n",
    "dataset['sentiment'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype \n",
      "---  ------     --------------  ----- \n",
      " 0   sentiment  25000 non-null  int64 \n",
      " 1   review     25000 non-null  object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 390.8+ KB\n"
     ]
    }
   ],
   "source": [
    "# and dropping the unique_id column\n",
    "dataset = dataset.dropna()\n",
    "dataset = dataset.drop(dataset.columns[0], axis=1)\n",
    "dataset.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing libraries that will aid in text cleaning and declaring regexes of items for removal\n",
    "import re\n",
    "import string\n",
    "\n",
    "html_tags = re.compile(r'<[^<]+?>')\n",
    "emojis = re.compile(\n",
    "    \"([\"\n",
    "    \"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
    "    \"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
    "    \"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "    \"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
    "    \"\\U0001F700-\\U0001F77F\"  # alchemical symbols\n",
    "    \"\\U0001F780-\\U0001F7FF\"  # Geometric Shapes Extended\n",
    "    \"\\U0001F800-\\U0001F8FF\"  # Supplemental Arrows-C\n",
    "    \"\\U0001F900-\\U0001F9FF\"  # Supplemental Symbols and Pictographs\n",
    "    \"\\U0001FA00-\\U0001FA6F\"  # Chess Symbols\n",
    "    \"\\U0001FA70-\\U0001FAFF\"  # Symbols and Pictographs Extended-A\n",
    "    \"\\U00002702-\\U000027B0\"  # Dingbats\n",
    "    \"])\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'them', \"needn't\", 'me', 'into', \"it's\", 'very', \"mustn't\", 'over', 'nor', 'yourself', 'didn', 'll', 'does', 'has', 'where', 'more', 'd', 'through', 'its', 'hadn', 'because', 'shan', 'hasn', 'in', \"you'd\", 'out', 'now', 'is', 'or', 'are', \"haven't\", 'these', 'this', \"doesn't\", 'from', 'haven', 'will', 'the', 'just', 'what', 'was', 'do', 'once', 'him', 'her', 'further', 'that', \"you've\", 'doing', 'so', 'such', 'but', \"aren't\", 'for', 'too', 'of', 'why', 'any', 'ourselves', 've', 'isn', 'whom', 'my', \"mightn't\", 'herself', \"shouldn't\", 're', 'it', 'no', \"isn't\", \"wouldn't\", \"hadn't\", \"you'll\", 'at', 'against', \"you're\", 'while', 'a', 'not', 'about', \"should've\", 'after', 'he', 'how', 'o', \"she's\", 'am', 'under', 'by', 'ours', 'here', 'than', 'if', \"didn't\", 'aren', 'she', 'shouldn', 'you', 'above', 'needn', 'own', 'who', 'all', \"don't\", 'himself', 'theirs', 'itself', \"that'll\", 'yours', 'there', 'each', 'had', 'should', 'having', 'won', 'which', 'mightn', 'then', 'mustn', 'when', 'to', 'weren', 'off', 'an', 'ma', 'again', \"weren't\", 'during', 'their', 'your', 'wouldn', 'were', 't', 'been', 'down', 'few', 'themselves', 'y', 'both', 'some', 'have', 'as', 'they', 'most', 'we', 'myself', 'being', 'and', 'i', 's', 'his', 'don', 'our', 'with', 'only', 'can', 'm', 'doesn', \"shan't\", 'those', 'couldn', \"wasn't\", 'until', 'wasn', 'between', 'ain', \"couldn't\", 'hers', 'be', 'on', \"won't\", 'did', 'below', 'other', 'yourselves', \"hasn't\", 'same', 'up', 'before'}\n"
     ]
    }
   ],
   "source": [
    "# import libraries that will allow for stopword removal, tokenization and lemmatization\n",
    "from nltk.tokenize import WordPunctTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer().lemmatize\n",
    "wordpunct_tokenize = WordPunctTokenizer().tokenize\n",
    "en_stop = set(stopwords.words('english'))\n",
    "print(en_stop)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>stuff going moment mj started listening music ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>classic war worlds timothy hines entertaining ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>film starts manager nicholas bell giving welco...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>must assumed praised film greatest filmed oper...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>superbly trashy wondrously unpretentious explo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                             review\n",
       "0          1  stuff going moment mj started listening music ...\n",
       "1          1  classic war worlds timothy hines entertaining ...\n",
       "2          0  film starts manager nicholas bell giving welco...\n",
       "3          0  must assumed praised film greatest filmed oper...\n",
       "4          1  superbly trashy wondrously unpretentious explo..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating a function that cleans the data takes a dataframe as an argument\n",
    "def cleaning_tool(dataframe):\n",
    "    '''\n",
    "    This function cleans the dataframe\n",
    "    '''\n",
    "    #removes html tags\n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: re.sub(html_tags, ' ', words)) \n",
    "    #removes emojis  \n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: re.sub(emojis, ' ', words))  \n",
    "    #removes digits [0-9]    \n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: ''.join([x for x in words if not x.isdigit()])) \n",
    "    #removes   non ASCII characters\n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: ''.join([x for x in words if x.isascii()]))\n",
    "    #converts text into lowercase  \n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: [str(words).lower()])\n",
    "    #does word tokenization  \n",
    "    dataframe['review'] = dataframe['review'].apply( lambda words: wordpunct_tokenize(str(words)))\n",
    "    #removes punctuation marks \n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: ''.join([str(words).translate( str.maketrans('', '', string.punctuation))]))\n",
    "    #word lemmatization   \n",
    "    dataframe['review'] = dataframe['review'].apply(lambda words: lemmatizer(str(words)))\n",
    "    # removes stopwords   \n",
    "    dataframe['review'] = dataframe['review'].apply( lambda words: ' '.join([x for x in str(words).split() if x not in (en_stop)]))\n",
    "       \n",
    "\n",
    "    return dataframe\n",
    "\n",
    "# applying the cleaning tool to the labelled data dataframe\n",
    "data = cleaning_tool(dataset)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Defining the dependent and independent variables\n",
    "Y = data['sentiment']\n",
    "X = data['review']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Text vectorization\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "reviews_vec = TfidfVectorizer()\n",
    "X_data = reviews_vec.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Columns: 73617 entries, aa to zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\n",
      "dtypes: float64(73617)\n",
      "memory usage: 13.7 GB\n"
     ]
    }
   ],
   "source": [
    "# creating a dataframe for use in feature comparison\n",
    "X_data_array = X_data.toarray()\n",
    "vocab = reviews_vec.get_feature_names_out()\n",
    "X_data_df = pd.DataFrame(X_data_array, columns=vocab)\n",
    "X_data_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# splitting the dataset into train and test data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    X_data, Y, test_size=0.20, random_state=12)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# building the model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()\n",
    "lr.fit(x_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#plot the sigmoid curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.887"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# obtaining the model accuracy\n",
    "lr.score(x_test, y_test)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2 : Scraping the Movie Reviews from Rotten Tomatoes and Feeding it into our Model for Sentiment Analysis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing beautifulsoup and requests library for webscraping\n",
    "from bs4 import BeautifulSoup\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# declaring necessary urls and setting up to web-scrape\n",
    "start_url = 'https://www.imdb.com/title/tt0816442/reviews?ref_=tt_urv'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Those familiar with the 2005 award winning and...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No extended fight scenes. No unnecessary pyrot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I do not know the book. But the film, for its ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Many books have been deemed 'unfilmable' - but...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"The Book Thief\" is certainly a rare kind of f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review\n",
       "0  Those familiar with the 2005 award winning and...\n",
       "1  No extended fight scenes. No unnecessary pyrot...\n",
       "2  I do not know the book. But the film, for its ...\n",
       "3  Many books have been deemed 'unfilmable' - but...\n",
       "4  \"The Book Thief\" is certainly a rare kind of f..."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# scraping reviews from the first page\n",
    "page = requests.get(start_url)\n",
    "soup = BeautifulSoup(page.text, \"html.parser\")\n",
    "list_of_reviews = [reviews.get_text() for reviews in soup.find_all(\n",
    "            'div', class_=\"text show-more__control\")]\n",
    "\n",
    "# appending data to dataframe\n",
    "columns = ['review']\n",
    "book_thief = pd.concat([pd.DataFrame([i], columns=columns)\n",
    "                  for i in list_of_reviews], ignore_index=True)\n",
    "book_thief.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>familiar award winning best selling novel aust...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>extended fight scenes unnecessary pyrotechnics...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>know book film beautiful simplicity useful hig...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>many books deemed unfilmable anyone read marku...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>book thief certainly rare kind film day gleams...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review\n",
       "0  familiar award winning best selling novel aust...\n",
       "1  extended fight scenes unnecessary pyrotechnics...\n",
       "2  know book film beautiful simplicity useful hig...\n",
       "3  many books deemed unfilmable anyone read marku...\n",
       "4  book thief certainly rare kind film day gleams..."
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cleaning the dataset\n",
    "book_thief = cleaning_tool(book_thief)\n",
    "book_thief.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorizing the reviews\n",
    "bookthief_revs = book_thief['review']\n",
    "reviews_vec = TfidfVectorizer()\n",
    "bookthief_data = reviews_vec.fit_transform(bookthief_revs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25 entries, 0 to 24\n",
      "Columns: 1926 entries, abandoned to zusak\n",
      "dtypes: float64(1926)\n",
      "memory usage: 376.3 KB\n"
     ]
    }
   ],
   "source": [
    "# creating a dataframe for feature comparison\n",
    "bookthief_data_array = bookthief_data.toarray()\n",
    "bookthiefvocab = reviews_vec.get_feature_names_out()\n",
    "bookthief_data_df = pd.DataFrame(bookthief_data_array, columns=bookthiefvocab)\n",
    "bookthief_data_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25 entries, 0 to 24\n",
      "Columns: 73617 entries, aa to zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\n",
      "dtypes: float64(73617)\n",
      "memory usage: 14.0 MB\n"
     ]
    }
   ],
   "source": [
    "# accounting for missing vocabulary\n",
    "not_exist_vocab = [v for v in X_data_df.columns.tolist()\n",
    "                   if v not in bookthief_data_df]\n",
    "bookthief_data_df = bookthief_data_df.reindex(\n",
    "    columns=bookthief_data_df.columns.tolist() + not_exist_vocab)\n",
    "bookthief_data_df = bookthief_data_df.fillna(0)\n",
    "bookthief_data_df = bookthief_data_df[X_data_df.columns.tolist()]\n",
    "bookthief_data_df.info()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting back to sparse matrix\n",
    "from scipy import sparse\n",
    "bookthief_data_array2 = bookthief_data_df.to_numpy()\n",
    "bookthief_data = sparse.csr_matrix(bookthief_data_array2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predicting the sentiment score\n",
    "predictedvalues = lr.predict(bookthief_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Positive Reviews : 21\n",
      "Number of Negative Reviews : 4\n"
     ]
    }
   ],
   "source": [
    "# Listing the number of reviews as classified by polarity\n",
    "print(f'Number of Positive Reviews : {list(predictedvalues).count(1)}')\n",
    "print(f'Number of Negative Reviews : {list(predictedvalues).count(0)}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ratings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ratings\n",
       "0       9\n",
       "1      10\n",
       "2       7\n",
       "3       8\n",
       "4      10"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Comparing with the actual ranking\n",
    "page = requests.get(start_url)\n",
    "soup = BeautifulSoup(page.text, \"html.parser\")\n",
    "list_of_ratings = [ratings.get_text() for ratings in soup.select('div.lister-item:nth-child(n) > div:nth-child(1) > div:nth-child(1) > div:nth-child(1) > span:nth-child(1) > span:nth-child(2)')]\n",
    "           \n",
    "\n",
    "# appending data to dataframe\n",
    "columns = ['ratings']\n",
    "book_thief_ratings = pd.concat([pd.DataFrame([i], columns=columns)\n",
    "                  for i in list_of_ratings], ignore_index=True)\n",
    "book_thief_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8     8\n",
       "9     5\n",
       "10    3\n",
       "7     3\n",
       "3     1\n",
       "6     1\n",
       "2     1\n",
       "5     1\n",
       "Name: ratings, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#determining the value counts of the score ratings\n",
    "book_thief_ratings.ratings.value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "As seen from the page, the ratings are done on a scale of 1 to 10, with values less than or equal to 5 being ranked negative, while those greater than 5 as positive\n",
    "From the page, 21 had a score of 6 and above, while 3 had a score of 5 and below.\n",
    "It is worth noting that two reviews did not give a rating score.\n",
    "In comparison, the model has done pretty well\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0873911fad79c748448388f34a7a5f37ba8a48b5927c4a6c38d28904f822843d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}