7 - Naive Bayes/Multinomial Naives Nayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4192c256",
   "metadata": {},
   "source": [
    "# Multinomial Naive Bayes\n",
    "\n",
    "## Overview\n",
    "- [1. Multinomial Naive Bayes](#1)\n",
    "- [2. Multinomial Naive Bayes Model from Scratch](#2)\n",
    "- [3. Multinomial Naive Bayes Model in `Sklearn`](#3)\n",
    "- [4. Multinomial Naive Bayes for `Out of Vocabulary`](#4)\n",
    "- [5. References](#5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1835deb",
   "metadata": {},
   "source": [
    "<a name='1' ></a>\n",
    "## 1. Multinomial Naive Bayes\n",
    "**Multinomial Naive Bayes** implements the naive Bayes algorithm for multinomially distributed data, and is one of the two classic naive Bayes variants used in text classification (where the data are typically represented as word vector counts, although tf-idf vectors are also known to work well in practice). \n",
    "\n",
    "Assume that we have pairs of data points $(w^{(i)}, y_i)$, where\n",
    "- $\\mathbf{w}^{(i)}$ is an observation (a text contain words)\n",
    "- $y_i$ is the label of $w^{(i)}$ \n",
    "\n",
    "for $i=1,2,...n$\n",
    "\n",
    "Construct a vocabulary from all texts $\\mathbf{w}^{(i)}$, we will get $V = \\{w_1, w_2, ..., w_d\\}$, $d$ is the number of distinct words. From $V$, $w^{(i)}$ present a $(1 \\times d)$ vector \n",
    "\n",
    "$$\\mathbf{w}^{(i)} = \\begin{bmatrix} N_{i1} & N_{i2} & \\dots & N_{id} \\end{bmatrix}$$\n",
    "\n",
    "where $N_{ij}$ is the frequency of word $j$ (or $w_j$) appears in the text $\\mathbf{w}^{(i)}$, $j=1,2,...,d$\n",
    "\n",
    "The probability that $\\mathbf{w}^{(i)}$ belongs to class $y=c$ is\n",
    "\n",
    "\\begin{split}\n",
    "\\begin{eqnarray}P(y=c|\\mathbf{w}^{(i)}) & = & \\frac{P(\\mathbf{w}^{(i)}| y=c) P(y=c)}{P(\\mathbf{w}^{(i)})} \\\\\n",
    "& \\propto & \\underbrace{P(y=c)}_{\\text{prior}} \\underbrace{\\prod_{j=1}^{d} P(w_j|y=c)^{N_{yj}}}_{\\text{likelihood}}\n",
    "\\end{eqnarray}\n",
    "\\end{split}\n",
    "\n",
    "Take the logarithm\n",
    "\n",
    "\\begin{align*}\n",
    "P(y=c|\\mathbf{w}^{(i)}) &\\propto \\log \\left(P(y=c) \\prod_{j=1}^{d} P(w_j|y=c)^{N_{yj}} \\right) \\\\\n",
    "&= \\log \\left( P(y=c) \\right) + \\sum_{j=1}^{d} N_{yj} \\log \\left( P(w_j|y=c) \\right)\n",
    "\\end{align*}\n",
    "\n",
    "We assign $\\theta_{yj}$ is the probability $P(w_j \\mid y=c)$ of feature $j$ (word $w_j$) appearing in a sample belonging to class $y=c$. From that, the distribution is parametrized by vectors $\\theta_y = (\\theta_{y1},\\ldots,\\theta_{yd})$  for each class $y=c$.\n",
    "\n",
    "The parameters $\\theta_{y}$  is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:\n",
    "\n",
    "$$\\hat{\\theta}_{yj} = \\frac{N_{yj}}{N_y}$$\n",
    "\n",
    "where $N_{yj} = \\sum_{\\mathbf{w} \\in T} w_j = \\sum_{i=1}^n N_{ij}$ is the number of times feature $j$  appears in a sample of class $y=c$  in the training set $T$, and $N_{y} = \\sum_{j=1}^{d} N_{yj}$ is the total count of all features for class $y=c$.\n",
    "\n",
    "However, if there is a word $w$ which doesn't appear in the training set with $y=c$, then $\\hat{\\theta}_{yj} = 0$ and $\\log (0)$ is unknown. To avoid, we maybe use\n",
    "\n",
    "$$\\hat{\\theta}_{yj} = \\frac{ N_{yj} + \\alpha}{N_j + \\alpha d}$$\n",
    "\n",
    "The smoothing priors $\\alpha \\geq 0$ accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting $\\alpha=1$ is called Laplace smoothing, while $\\alpha < 1$  is called Lidstone smoothing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69e7ed07",
   "metadata": {},
   "source": [
    "<a name='2' ></a>\n",
    "## 2. Multinomial Naive Bayes Model from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d955216e",
   "metadata": {},
   "source": [
    "### Import package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1e6a99a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3cd1050",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9ba6ff98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11314, 130107)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dataset\n",
    "X, y = datasets.fetch_20newsgroups(return_X_y=True)\n",
    "\n",
    "count_vect = CountVectorizer()\n",
    "X_counts = count_vect.fit_transform(X)\n",
    "X_counts.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39d6236",
   "metadata": {},
   "source": [
    "### Build model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8c842bb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MNaiveBayes():\n",
    "    \"\"\"Build Multinomial Naive Bayes from scratch.\"\"\"\n",
    "    \n",
    "    def __init__(self, alpha=1):\n",
    "        self.alpha = alpha\n",
    "        self.log_prior = None\n",
    "        self.log_prob = None     # log likelihood\n",
    "        self.classes = None     # unique label\n",
    "        self.class_count = None     # count each unique label\n",
    "        self.n_features = None    # number of distinct words (V)\n",
    "        self.n_samples = None    # number of observations (text)\n",
    "        \n",
    "    def _likelihood(self, freqs):\n",
    "        \"\"\"Calculate likelihood of probabilites P(w_i | y).\"\"\"\n",
    "        lmbda = (freqs + self.alpha) / (np.sum(freqs) + self.n_features * self.alpha)\n",
    "        return lmbda\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        \"\"\"\n",
    "        Traing model to find log_prior and log_prob (or log likelihood).\n",
    "        \n",
    "        Args:\n",
    "            X: (n_samples, n_features) array, each sample is a text\n",
    "            y: (n_samples, ) labels correspond to samples\n",
    "        \n",
    "        \"\"\"\n",
    "        self.classes, self.class_count = np.unique(y, return_counts=True)\n",
    "        self.n_samples, self.n_features = X.shape\n",
    "        \n",
    "        # Calculate log prior for P(y=c)\n",
    "        self.log_prior = np.log(self.class_count / self.n_samples)\n",
    "        \n",
    "        # Calculate log likelihood for each class in y\n",
    "        self.log_prob = []\n",
    "        for label in self.classes:\n",
    "            indices = np.where(y == label)[0]\n",
    "            freqs = np.sum(X[indices, :], axis=0)\n",
    "            log_prob_label = np.log(self._likelihood(freqs))\n",
    "            self.log_prob.append(log_prob_label)\n",
    "            \n",
    "        self.log_prob = np.array(self.log_prob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3e775686",
   "metadata": {},
   "outputs": [],
   "source": [
    "mnb = MNaiveBayes()\n",
    "mnb.fit(X_counts, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7d8149d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-3.16001007, -2.96389519, -2.95198016, -2.95367364, -2.97422231,\n",
       "       -2.94860178, -2.96218433, -2.94691686, -2.94020542, -2.94187906,\n",
       "       -2.93686652, -2.94523477, -2.95198016, -2.94691686, -2.94860178,\n",
       "       -2.93853458, -3.0311772 , -2.99874192, -3.19175877, -3.40155099])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnb.log_prior"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "38698007",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[-11.20532976,  -9.03627606, -12.59162412, ..., -12.59162412,\n",
       "         -12.59162412, -12.59162412]],\n",
       "\n",
       "       [[ -8.92151479,  -9.58649109, -12.47686285, ..., -12.47686285,\n",
       "         -12.47686285, -12.47686285]],\n",
       "\n",
       "       [[ -9.40038692, -10.52885217, -12.92674744, ..., -12.92674744,\n",
       "         -12.92674744, -12.92674744]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[ -9.38895408,  -7.58259581, -12.22216743, ..., -12.91531461,\n",
       "         -12.91531461, -12.91531461]],\n",
       "\n",
       "       [[ -9.35181599,  -8.45643194, -12.71911182, ..., -12.71911182,\n",
       "         -12.71911182, -12.71911182]],\n",
       "\n",
       "       [[-10.86655118,  -9.58561733, -12.47598909, ..., -12.47598909,\n",
       "         -12.47598909, -12.47598909]]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnb.log_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "149f794f",
   "metadata": {},
   "source": [
    "<a name='3' ></a>\n",
    "## 3. Multinomial Naive Bayes Model in `sklearn`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "47b7c3a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>MultinomialNB()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MultinomialNB</label><div class=\"sk-toggleable__content\"><pre>MultinomialNB()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "MultinomialNB()"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB()\n",
    "clf.fit(X_counts, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3f2d8948",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-3.16001007, -2.96389519, -2.95198016, -2.95367364, -2.97422231,\n",
       "       -2.94860178, -2.96218433, -2.94691686, -2.94020542, -2.94187906,\n",
       "       -2.93686652, -2.94523477, -2.95198016, -2.94691686, -2.94860178,\n",
       "       -2.93853458, -3.0311772 , -2.99874192, -3.19175877, -3.40155099])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.class_log_prior_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "76696d2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-11.20532976,  -9.03627606, -12.59162412, ..., -12.59162412,\n",
       "        -12.59162412, -12.59162412],\n",
       "       [ -8.92151479,  -9.58649109, -12.47686285, ..., -12.47686285,\n",
       "        -12.47686285, -12.47686285],\n",
       "       [ -9.40038692, -10.52885217, -12.92674744, ..., -12.92674744,\n",
       "        -12.92674744, -12.92674744],\n",
       "       ...,\n",
       "       [ -9.38895408,  -7.58259581, -12.22216743, ..., -12.91531461,\n",
       "        -12.91531461, -12.91531461],\n",
       "       [ -9.35181599,  -8.45643194, -12.71911182, ..., -12.71911182,\n",
       "        -12.71911182, -12.71911182],\n",
       "       [-10.86655118,  -9.58561733, -12.47598909, ..., -12.47598909,\n",
       "        -12.47598909, -12.47598909]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.feature_log_prob_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5960070c",
   "metadata": {},
   "source": [
    "**Nice! It's easy to see that our model and sklearn's model share the common result! Are you wondering that we haven't written `predict` for our model? The answer is in the below part.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f3ca17",
   "metadata": {},
   "source": [
    "<a name='4' ></a>\n",
    "## 4. Multinomial Naive Bayes for `Out of Vocabulary`\n",
    "Go into detail, you read `predict funtion` of `Multinomial Naive Bayes` in [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB). It demands a vector `(1, n_features)`, where `n_features` is the number of appeared distinct words. However, if you have a new dataset for testing, have words not appearing in the training dataset (They is called `out of vocabulary`). How to get the presented vector to predict?\n",
    "\n",
    "So we need to custom our model to face the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "bf6a8f18",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MNaiveBayesV2():\n",
    "    def __init__(self, alpha=1):\n",
    "        self.alpha = alpha\n",
    "        self.log_priors = None\n",
    "        self.log_likelihoods = None\n",
    "        self.labels_count = None\n",
    "        self.vocab = None\n",
    "        self.V = None     # length of vocabulary (d)\n",
    "        self.freqs = None\n",
    "        self.label_list = None\n",
    "        \n",
    "    def _process(self, sentence):\n",
    "        \"\"\"Split a sentence into a list.\"\"\"\n",
    "        return sentence.split(' ')\n",
    "    \n",
    "    def _get_freqs(self, data, label):\n",
    "        \"\"\"Count frequency of each word in a label.\"\"\"\n",
    "        freqs = {}\n",
    "        for sentence, label in zip(data, label):\n",
    "            for word in self._process(sentence):\n",
    "                # define key of dict\n",
    "                pair = (word, label)\n",
    "                # If the key exists in the dictionary, increment to 1\n",
    "                if pair in freqs:\n",
    "                    freqs[pair] += 1        \n",
    "                # else, if the key new, set equal to 1\n",
    "                else:\n",
    "                    freqs[pair] = 1\n",
    "        return freqs\n",
    "    \n",
    "    def _labels_count(self, label):\n",
    "        \"\"\"Calculate the number of words depending on label y=c.\"\"\"\n",
    "        Ny = 0\n",
    "        for pair, freq in self.freqs.items():\n",
    "            if pair[1] == label:\n",
    "                Ny += freq\n",
    "        return Ny\n",
    "    \n",
    "    def _likelihood(self, Nyj, Ny):\n",
    "        \"\"\"\n",
    "        Laplace smoothing formula to compute probability of word w_i belong to class c\n",
    "        \"\"\"\n",
    "        lmbda = (Nyj + self.alpha) / (Ny + self.V * self.alpha)\n",
    "        return lmbda \n",
    "\n",
    "    def _lookup(self, word, label):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            freqs: a dictionary with the frequency of each pair (or tuple)\n",
    "            word: the word to look up\n",
    "            label: the label corresponding to the word\n",
    "        Return:\n",
    "            the number of times the word with its corresponding label appearing.\n",
    "        \"\"\"\n",
    "        return self.freqs.get((word, label), 0)\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        \"\"\"\n",
    "        Training model to find parameters.\n",
    "        \n",
    "        Args:\n",
    "            X: a list or array contains all traing texts\n",
    "            y: label for each text in X\n",
    "        \"\"\"\n",
    "        # calculate log prior P(y=c)\n",
    "        n_samples = len(y)\n",
    "        self.label_list, freqs = np.unique(y, return_counts=True)\n",
    "        self.log_priors = {label: np.log(freq / n_samples) \n",
    "                           for label, freq in zip(self.label_list, freqs)}\n",
    "\n",
    "        # get labels_count\n",
    "        self.freqs = self._get_freqs(X, y)\n",
    "        self.labels_count = {label: self._labels_count(label) for label in self.label_list}\n",
    "\n",
    "        # calculate likelihood\n",
    "        self.log_likelihoods = {}\n",
    "        self.vocab = set([pair[0] for pair in self.freqs.keys()])\n",
    "        self.V = len(self.vocab)\n",
    "        for label in self.label_list:\n",
    "            # total words of label c\n",
    "            Ny = self.labels_count[label]\n",
    "            for word in self.vocab:\n",
    "                # total word w_i in label c\n",
    "                Nyj = self._lookup(word, label)\n",
    "                # likelihood P(wi|y=c)^Nyj\n",
    "                theta = np.log(self._likelihood(Nyj, Ny))\n",
    "                self.log_likelihoods[(word, label)] = Nyj * theta\n",
    "                \n",
    "                \n",
    "    def predict(self, sentence):\n",
    "        \"\"\"Predict the label for input text.\"\"\"\n",
    "        word_list = self._process(sentence)\n",
    "        word_set = set(word_list)\n",
    "        score_list = []\n",
    "        \n",
    "        # choose label with the maximum probability\n",
    "        for label in self.label_list:\n",
    "            prob = self.log_priors[label]\n",
    "            for word in word_set:\n",
    "                if word in self.vocab:\n",
    "                    log_likelihood = self.log_likelihoods[(word, label)]\n",
    "                    prob += log_likelihood\n",
    "\n",
    "            score_list.append(prob)\n",
    "        return np.argmax(score_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "001e4001",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: street food, a traditional culture of Vietnamese people - Label: 0\n",
      "Sentence: Messi has been acquired by a football team sport in France - Label: 1\n"
     ]
    }
   ],
   "source": [
    "# Initialize dataset and train model\n",
    "sentence1 = 'Pho is one of the traditional food on the Hanois street'\n",
    "sentence2 = 'taste of roasted duck creates a traditional food'\n",
    "sentence3 = 'football is the king sport sport sport sport'\n",
    "sentence4 = 'world cup is the biggest football football football festival on the planet'\n",
    "\n",
    "X_train = [sentence1, sentence2, sentence3, sentence4]\n",
    "y_train = [0, 0, 1, 1]\n",
    "\n",
    "# Initialize model\n",
    "mnbv2 = MNaiveBayesV2()\n",
    "mnbv2.fit(X_train, y_train)\n",
    "\n",
    "# # Test model\n",
    "X_test = ['street food, a traditional culture of Vietnamese people', \n",
    "          'Messi has been acquired by a football team sport in France']\n",
    "\n",
    "for sentence in X_test:\n",
    "    y_pred = mnbv2.predict(sentence)\n",
    "    print(f'Sentence: {sentence} - Label: {y_pred}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c480b383",
   "metadata": {},
   "source": [
    "<a name='5' ></a>\n",
    "## 5. References\n",
    "\n",
    "- [https://en.wikipedia.org/wiki/Naive_Bayes_classifier](https://en.wikipedia.org/wiki/Naive_Bayes_classifier)\n",
    "- [https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case](https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case)\n",
    "- [https://machinelearningcoban.com/2017/08/08/nbc/](https://machinelearningcoban.com/2017/08/08/nbc/)\n",
    "- [https://scikit-learn.org/stable/modules/naive_bayes.html#](https://scikit-learn.org/stable/modules/naive_bayes.html#)\n",
    "- [https://www.python-engineer.com/courses/mlfromscratch/05_naivebayes/](https://www.python-engineer.com/courses/mlfromscratch/05_naivebayes/)\n",
    "- [https://phamdinhkhanh.github.io/deepai-book/ch_ml/NaiveBayes.html](https://phamdinhkhanh.github.io/deepai-book/ch_ml/NaiveBayes.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}